Note: This is test shard 8 of 8.
[==========] Running 2 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AllTypesItest/3, where TypeParam = kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)7> >
[ RUN ] AllTypesItest/3.TestTimestampPadding
2026-08-09T19:58:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-09T19:58:11Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260809 19:58:11.429747 3296 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskmEiWMh/build/debug/bin/kudu
/tmp/dist-test-taskmEiWMh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.56.62:33635
--webserver_interface=127.3.56.62
--webserver_port=0
--builtin_ntp_servers=127.3.56.20:44297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.56.62:33635 with env {}
W20260809 19:58:11.531786 3307 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 19:58:11.531987 3307 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 19:58:11.532023 3307 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260809 19:58:11.535327 3307 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 19:58:11.535377 3307 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260809 19:58:11.535396 3307 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 19:58:11.535413 3307 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260809 19:58:11.539125 3307 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.56.20:44297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.56.62:33635
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.56.62:33635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.56.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.3307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
build type DEBUG
built by None at 09 Aug 2026 19:43:14 UTC on bdcb31816ec0
build id 13260
I20260809 19:58:11.540020 3307 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 19:58:11.541016 3307 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260809 19:58:11.546540 3307 server_base.cc:1061] running on GCE node
W20260809 19:58:11.546469 3317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 19:58:11.546469 3315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 19:58:11.546487 3314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 19:58:11.547228 3307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 19:58:11.548055 3307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 19:58:11.549237 3307 hybrid_clock.cc:648] HybridClock initialized: now 1786305491549228 us; error 112 us; skew 500 ppm
I20260809 19:58:11.550957 3307 webserver.cc:533] Webserver started at http://127.3.56.62:42495/ using document root <none> and password file <none>
I20260809 19:58:11.551481 3307 fs_manager.cc:362] Metadata directory not provided
I20260809 19:58:11.551566 3307 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 19:58:11.551781 3307 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 19:58:11.553238 3307 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/instance:
uuid: "8b4ad58fdc334ba99ab671d1f7a6f422"
format_stamp: "Formatted at 2026-08-09 19:58:11 on dist-test-slave-bw88"
I20260809 19:58:11.553799 3307 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/wal/instance:
uuid: "8b4ad58fdc334ba99ab671d1f7a6f422"
format_stamp: "Formatted at 2026-08-09 19:58:11 on dist-test-slave-bw88"
I20260809 19:58:11.556778 3307 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 19:58:11.558856 3324 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 19:58:11.559798 3307 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260809 19:58:11.559930 3307 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/wal
uuid: "8b4ad58fdc334ba99ab671d1f7a6f422"
format_stamp: "Formatted at 2026-08-09 19:58:11 on dist-test-slave-bw88"
I20260809 19:58:11.560037 3307 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 19:58:11.577894 3307 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260809 19:58:11.578562 3307 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 19:58:11.578768 3307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 19:58:11.586071 3395 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.56.62:33635 every 8 connection(s)
I20260809 19:58:11.586078 3307 rpc_server.cc:307] RPC server started. Bound to: 127.3.56.62:33635
I20260809 19:58:11.587103 3307 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb
I20260809 19:58:11.589823 3397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 19:58:11.594565 3397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422: Bootstrap starting.
I20260809 19:58:11.595556 3296 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskmEiWMh/build/debug/bin/kudu as pid 3307
I20260809 19:58:11.595731 3296 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/wal/instance
I20260809 19:58:11.597399 3397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422: Neither blocks nor log segments found. Creating new log.
I20260809 19:58:11.598268 3397 log.cc:826] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422: Log is configured to *not* fsync() on all Append() calls
I20260809 19:58:11.600250 3397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422: No bootstrap required, opened a new log
I20260809 19:58:11.602933 3397 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b4ad58fdc334ba99ab671d1f7a6f422" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 33635 } }
I20260809 19:58:11.603098 3397 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 19:58:11.603164 3397 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b4ad58fdc334ba99ab671d1f7a6f422, State: Initialized, Role: FOLLOWER
I20260809 19:58:11.603614 3397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b4ad58fdc334ba99ab671d1f7a6f422" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 33635 } }
I20260809 19:58:11.603760 3397 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 19:58:11.603842 3397 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 19:58:11.603951 3397 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422 [term 0 FOLLOWER]: Advancing to term 1
I20260809 19:58:11.604753 3397 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b4ad58fdc334ba99ab671d1f7a6f422" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 33635 } }
I20260809 19:58:11.605077 3397 leader_election.cc:304] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b4ad58fdc334ba99ab671d1f7a6f422; no voters:
I20260809 19:58:11.605391 3397 leader_election.cc:290] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 19:58:11.605491 3405 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 19:58:11.605697 3405 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422 [term 1 LEADER]: Becoming Leader. State: Replica: 8b4ad58fdc334ba99ab671d1f7a6f422, State: Running, Role: LEADER
I20260809 19:58:11.606014 3405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b4ad58fdc334ba99ab671d1f7a6f422" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 33635 } }
I20260809 19:58:11.606547 3397 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 19:58:11.607569 3407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b4ad58fdc334ba99ab671d1f7a6f422" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b4ad58fdc334ba99ab671d1f7a6f422" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 33635 } } }
I20260809 19:58:11.607695 3407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422 [sys.catalog]: This master's current role is: LEADER
I20260809 19:58:11.608086 3406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b4ad58fdc334ba99ab671d1f7a6f422. Latest consensus state: current_term: 1 leader_uuid: "8b4ad58fdc334ba99ab671d1f7a6f422" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b4ad58fdc334ba99ab671d1f7a6f422" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 33635 } } }
I20260809 19:58:11.609367 3406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422 [sys.catalog]: This master's current role is: LEADER
W20260809 19:58:11.611728 3426 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 19:58:11.611860 3426 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 19:58:11.611977 3415 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260809 19:58:11.612913 3415 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260809 19:58:11.617892 3415 catalog_manager.cc:1383] Generated new cluster ID: 3f907c7c44c0491c9b386b5360e6d676
I20260809 19:58:11.617971 3415 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260809 19:58:11.643020 3415 catalog_manager.cc:1406] Generated new certificate authority record
I20260809 19:58:11.643842 3415 catalog_manager.cc:1540] Loading token signing keys...
I20260809 19:58:11.655064 3415 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422: Generated new TSK 0
I20260809 19:58:11.655723 3415 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260809 19:58:11.665482 3296 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskmEiWMh/build/debug/bin/kudu
/tmp/dist-test-taskmEiWMh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.56.1:0
--local_ip_for_outbound_sockets=127.3.56.1
--webserver_interface=127.3.56.1
--webserver_port=0
--tserver_master_addrs=127.3.56.62:33635
--builtin_ntp_servers=127.3.56.20:44297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
W20260809 19:58:11.770362 3432 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 19:58:11.770612 3432 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 19:58:11.770664 3432 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260809 19:58:11.773897 3432 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260809 19:58:11.773967 3432 flags.cc:432] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260809 19:58:11.773985 3432 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=1
W20260809 19:58:11.774060 3432 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.56.1
I20260809 19:58:11.777835 3432 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.56.20:44297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.56.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.56.1
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.3.56.62:33635
--never_fsync=true
--heap_profile_path=/tmp/kudu.3432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.56.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
build type DEBUG
built by None at 09 Aug 2026 19:43:14 UTC on bdcb31816ec0
build id 13260
I20260809 19:58:11.778813 3432 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 19:58:11.779870 3432 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260809 19:58:11.786002 3437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 19:58:11.786011 3438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 19:58:11.786047 3440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 19:58:11.786275 3432 server_base.cc:1061] running on GCE node
I20260809 19:58:11.786660 3432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 19:58:11.787189 3432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 19:58:11.788401 3432 hybrid_clock.cc:648] HybridClock initialized: now 1786305491788377 us; error 39 us; skew 500 ppm
I20260809 19:58:11.790233 3432 webserver.cc:533] Webserver started at http://127.3.56.1:37481/ using document root <none> and password file <none>
I20260809 19:58:11.790783 3432 fs_manager.cc:362] Metadata directory not provided
I20260809 19:58:11.790875 3432 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 19:58:11.791100 3432 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 19:58:11.792604 3432 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-0/data/instance:
uuid: "58b4bccc13254c8988cf60c89783f89a"
format_stamp: "Formatted at 2026-08-09 19:58:11 on dist-test-slave-bw88"
I20260809 19:58:11.793095 3432 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-0/wal/instance:
uuid: "58b4bccc13254c8988cf60c89783f89a"
format_stamp: "Formatted at 2026-08-09 19:58:11 on dist-test-slave-bw88"
I20260809 19:58:11.796319 3432 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 19:58:11.798533 3450 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 19:58:11.799410 3432 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 19:58:11.799567 3432 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-0/wal
uuid: "58b4bccc13254c8988cf60c89783f89a"
format_stamp: "Formatted at 2026-08-09 19:58:11 on dist-test-slave-bw88"
I20260809 19:58:11.799667 3432 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 19:58:11.831763 3432 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260809 19:58:11.832437 3432 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 19:58:11.832631 3432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 19:58:11.833161 3432 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260809 19:58:11.834154 3432 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 19:58:11.834203 3432 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 19:58:11.834272 3432 ts_tablet_manager.cc:616] Registered 0 tablets
I20260809 19:58:11.834303 3432 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 19:58:11.842759 3432 rpc_server.cc:307] RPC server started. Bound to: 127.3.56.1:37591
I20260809 19:58:11.842837 3601 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.56.1:37591 every 8 connection(s)
I20260809 19:58:11.843694 3432 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-0/data/info.pb
I20260809 19:58:11.851742 3296 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskmEiWMh/build/debug/bin/kudu as pid 3432
I20260809 19:58:11.851882 3296 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-0/wal/instance
I20260809 19:58:11.852885 3602 heartbeater.cc:344] Connected to a master server at 127.3.56.62:33635
I20260809 19:58:11.853127 3602 heartbeater.cc:461] Registering TS with master...
I20260809 19:58:11.853683 3602 heartbeater.cc:507] Master 127.3.56.62:33635 requested a full tablet report, sending...
I20260809 19:58:11.854120 3296 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskmEiWMh/build/debug/bin/kudu
/tmp/dist-test-taskmEiWMh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.56.2:0
--local_ip_for_outbound_sockets=127.3.56.2
--webserver_interface=127.3.56.2
--webserver_port=0
--tserver_master_addrs=127.3.56.62:33635
--builtin_ntp_servers=127.3.56.20:44297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
I20260809 19:58:11.855159 3345 ts_manager.cc:194] Registered new tserver with Master: 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591)
I20260809 19:58:11.856451 3345 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.56.1:47731
W20260809 19:58:11.957954 3610 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 19:58:11.958156 3610 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 19:58:11.958206 3610 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260809 19:58:11.961772 3610 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260809 19:58:11.961858 3610 flags.cc:432] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260809 19:58:11.961879 3610 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=1
W20260809 19:58:11.962003 3610 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.56.2
I20260809 19:58:11.968271 3610 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.56.20:44297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.56.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.56.2
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.3.56.62:33635
--never_fsync=true
--heap_profile_path=/tmp/kudu.3610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.56.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
build type DEBUG
built by None at 09 Aug 2026 19:43:14 UTC on bdcb31816ec0
build id 13260
I20260809 19:58:11.969259 3610 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 19:58:11.970459 3610 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260809 19:58:11.976859 3616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 19:58:11.976842 3617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 19:58:11.976933 3619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 19:58:11.977054 3610 server_base.cc:1061] running on GCE node
I20260809 19:58:11.977568 3610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 19:58:11.978083 3610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 19:58:11.979255 3610 hybrid_clock.cc:648] HybridClock initialized: now 1786305491979241 us; error 33 us; skew 500 ppm
I20260809 19:58:11.981084 3610 webserver.cc:533] Webserver started at http://127.3.56.2:40387/ using document root <none> and password file <none>
I20260809 19:58:11.981696 3610 fs_manager.cc:362] Metadata directory not provided
I20260809 19:58:11.981778 3610 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 19:58:11.981971 3610 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 19:58:11.983449 3610 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/instance:
uuid: "20da41f8404d40e59d12b8d87f881a1d"
format_stamp: "Formatted at 2026-08-09 19:58:11 on dist-test-slave-bw88"
I20260809 19:58:11.983919 3610 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/wal/instance:
uuid: "20da41f8404d40e59d12b8d87f881a1d"
format_stamp: "Formatted at 2026-08-09 19:58:11 on dist-test-slave-bw88"
I20260809 19:58:11.986969 3610 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 19:58:11.988931 3627 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 19:58:11.989848 3610 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 19:58:11.989989 3610 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/wal
uuid: "20da41f8404d40e59d12b8d87f881a1d"
format_stamp: "Formatted at 2026-08-09 19:58:11 on dist-test-slave-bw88"
I20260809 19:58:11.990087 3610 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 19:58:12.009972 3610 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260809 19:58:12.010578 3610 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 19:58:12.010763 3610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 19:58:12.011268 3610 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260809 19:58:12.012182 3610 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 19:58:12.012250 3610 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 19:58:12.012316 3610 ts_tablet_manager.cc:616] Registered 0 tablets
I20260809 19:58:12.012358 3610 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 19:58:12.021536 3610 rpc_server.cc:307] RPC server started. Bound to: 127.3.56.2:34843
I20260809 19:58:12.021584 3774 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.56.2:34843 every 8 connection(s)
I20260809 19:58:12.022424 3610 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb
I20260809 19:58:12.030606 3775 heartbeater.cc:344] Connected to a master server at 127.3.56.62:33635
I20260809 19:58:12.030814 3296 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskmEiWMh/build/debug/bin/kudu as pid 3610
I20260809 19:58:12.030834 3775 heartbeater.cc:461] Registering TS with master...
I20260809 19:58:12.030902 3296 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/wal/instance
I20260809 19:58:12.031324 3775 heartbeater.cc:507] Master 127.3.56.62:33635 requested a full tablet report, sending...
I20260809 19:58:12.032434 3344 ts_manager.cc:194] Registered new tserver with Master: 20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843)
I20260809 19:58:12.033033 3344 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.56.2:57017
I20260809 19:58:12.033227 3296 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskmEiWMh/build/debug/bin/kudu
/tmp/dist-test-taskmEiWMh/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.56.3:0
--local_ip_for_outbound_sockets=127.3.56.3
--webserver_interface=127.3.56.3
--webserver_port=0
--tserver_master_addrs=127.3.56.62:33635
--builtin_ntp_servers=127.3.56.20:44297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
W20260809 19:58:12.135282 3779 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 19:58:12.135475 3779 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 19:58:12.135524 3779 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260809 19:58:12.138873 3779 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260809 19:58:12.138937 3779 flags.cc:432] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260809 19:58:12.138954 3779 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=1
W20260809 19:58:12.139034 3779 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.56.3
I20260809 19:58:12.142791 3779 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.56.20:44297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.56.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.56.3
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.3.56.62:33635
--never_fsync=true
--heap_profile_path=/tmp/kudu.3779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.56.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
build type DEBUG
built by None at 09 Aug 2026 19:43:14 UTC on bdcb31816ec0
build id 13260
I20260809 19:58:12.143754 3779 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 19:58:12.144786 3779 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260809 19:58:12.150924 3785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 19:58:12.150924 3787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 19:58:12.150964 3790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 19:58:12.151028 3779 server_base.cc:1061] running on GCE node
I20260809 19:58:12.151448 3779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 19:58:12.151955 3779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 19:58:12.153138 3779 hybrid_clock.cc:648] HybridClock initialized: now 1786305492153126 us; error 29 us; skew 500 ppm
I20260809 19:58:12.154884 3779 webserver.cc:533] Webserver started at http://127.3.56.3:40625/ using document root <none> and password file <none>
I20260809 19:58:12.155424 3779 fs_manager.cc:362] Metadata directory not provided
I20260809 19:58:12.155505 3779 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 19:58:12.155689 3779 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 19:58:12.157151 3779 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/instance:
uuid: "badc9e1d2ce547869d5d9ebf74a45b74"
format_stamp: "Formatted at 2026-08-09 19:58:12 on dist-test-slave-bw88"
I20260809 19:58:12.157739 3779 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/wal/instance:
uuid: "badc9e1d2ce547869d5d9ebf74a45b74"
format_stamp: "Formatted at 2026-08-09 19:58:12 on dist-test-slave-bw88"
I20260809 19:58:12.160779 3779 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 19:58:12.162928 3798 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 19:58:12.163792 3779 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 19:58:12.163919 3779 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/wal
uuid: "badc9e1d2ce547869d5d9ebf74a45b74"
format_stamp: "Formatted at 2026-08-09 19:58:12 on dist-test-slave-bw88"
I20260809 19:58:12.164013 3779 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 19:58:12.177237 3779 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260809 19:58:12.177779 3779 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 19:58:12.177955 3779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 19:58:12.178450 3779 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260809 19:58:12.179316 3779 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 19:58:12.179385 3779 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 19:58:12.179448 3779 ts_tablet_manager.cc:616] Registered 0 tablets
I20260809 19:58:12.179494 3779 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 19:58:12.188724 3779 rpc_server.cc:307] RPC server started. Bound to: 127.3.56.3:46687
I20260809 19:58:12.188763 3959 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.56.3:46687 every 8 connection(s)
I20260809 19:58:12.189677 3779 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb
I20260809 19:58:12.198546 3960 heartbeater.cc:344] Connected to a master server at 127.3.56.62:33635
I20260809 19:58:12.198848 3960 heartbeater.cc:461] Registering TS with master...
I20260809 19:58:12.199177 3296 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskmEiWMh/build/debug/bin/kudu as pid 3779
I20260809 19:58:12.199252 3296 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/wal/instance
I20260809 19:58:12.199424 3960 heartbeater.cc:507] Master 127.3.56.62:33635 requested a full tablet report, sending...
I20260809 19:58:12.200637 3345 ts_manager.cc:194] Registered new tserver with Master: badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687)
I20260809 19:58:12.201244 3345 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.56.3:59811
I20260809 19:58:12.201963 3296 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260809 19:58:12.212734 3345 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50948:
name: "all-types-table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int8_val"
type: INT8
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int16_val"
type: INT16
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int32_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int64_val"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "timestamp_val"
type: UNIXTIME_MICROS
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "date_val"
type: DATE
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "varchar_val"
type: VARCHAR
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
length: 65535
}
immutable: false
}
columns {
name: "bool_val"
type: BOOL
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "float_val"
type: FLOAT
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "double_val"
type: DOUBLE
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "binary_val"
type: BINARY
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "decimal32_val"
type: DECIMAL32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 9
scale: 0
}
immutable: false
}
columns {
name: "decimal64_val"
type: DECIMAL64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 18
scale: 0
}
immutable: false
}
columns {
name: "decimal128_val"
type: DECIMAL128
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 38
scale: 0
}
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\000\000\252\252\252*\000\000\000\000\004\001\000\000\000TUUU\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 19:58:12.215603 3345 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table all-types-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 19:58:12.233301 3867 tablet_service.cc:1511] Processing CreateTablet for tablet 5814361c2f24496fa296e86169236f87 (DEFAULT_TABLE table=all-types-table [id=82e6272836bc4c2ca0159253dfc97038]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20260809 19:58:12.233201 3692 tablet_service.cc:1511] Processing CreateTablet for tablet 97d3cd24f4b24b578165968c3aa82513 (DEFAULT_TABLE table=all-types-table [id=82e6272836bc4c2ca0159253dfc97038]), partition=RANGE (key) PARTITION VALUES < 715827882
I20260809 19:58:12.233867 3690 tablet_service.cc:1511] Processing CreateTablet for tablet 5814361c2f24496fa296e86169236f87 (DEFAULT_TABLE table=all-types-table [id=82e6272836bc4c2ca0159253dfc97038]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20260809 19:58:12.233572 3868 tablet_service.cc:1511] Processing CreateTablet for tablet 8375821116104cf999ae49469e06def0 (DEFAULT_TABLE table=all-types-table [id=82e6272836bc4c2ca0159253dfc97038]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20260809 19:58:12.234514 3867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5814361c2f24496fa296e86169236f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 19:58:12.234514 3692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97d3cd24f4b24b578165968c3aa82513. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 19:58:12.234589 3516 tablet_service.cc:1511] Processing CreateTablet for tablet 97d3cd24f4b24b578165968c3aa82513 (DEFAULT_TABLE table=all-types-table [id=82e6272836bc4c2ca0159253dfc97038]), partition=RANGE (key) PARTITION VALUES < 715827882
I20260809 19:58:12.235203 3515 tablet_service.cc:1511] Processing CreateTablet for tablet 8375821116104cf999ae49469e06def0 (DEFAULT_TABLE table=all-types-table [id=82e6272836bc4c2ca0159253dfc97038]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20260809 19:58:12.235723 3515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8375821116104cf999ae49469e06def0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 19:58:12.234694 3510 tablet_service.cc:1511] Processing CreateTablet for tablet 5814361c2f24496fa296e86169236f87 (DEFAULT_TABLE table=all-types-table [id=82e6272836bc4c2ca0159253dfc97038]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20260809 19:58:12.237514 3690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5814361c2f24496fa296e86169236f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 19:58:12.237659 3510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5814361c2f24496fa296e86169236f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 19:58:12.239127 3868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8375821116104cf999ae49469e06def0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 19:58:12.241400 3516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97d3cd24f4b24b578165968c3aa82513. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 19:58:12.233208 3691 tablet_service.cc:1511] Processing CreateTablet for tablet 8375821116104cf999ae49469e06def0 (DEFAULT_TABLE table=all-types-table [id=82e6272836bc4c2ca0159253dfc97038]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20260809 19:58:12.241874 3691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8375821116104cf999ae49469e06def0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 19:58:12.241906 3869 tablet_service.cc:1511] Processing CreateTablet for tablet 97d3cd24f4b24b578165968c3aa82513 (DEFAULT_TABLE table=all-types-table [id=82e6272836bc4c2ca0159253dfc97038]), partition=RANGE (key) PARTITION VALUES < 715827882
I20260809 19:58:12.242168 3869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97d3cd24f4b24b578165968c3aa82513. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 19:58:12.243537 3985 tablet_bootstrap.cc:492] T 5814361c2f24496fa296e86169236f87 P 58b4bccc13254c8988cf60c89783f89a: Bootstrap starting.
I20260809 19:58:12.244475 3983 tablet_bootstrap.cc:492] T 5814361c2f24496fa296e86169236f87 P badc9e1d2ce547869d5d9ebf74a45b74: Bootstrap starting.
I20260809 19:58:12.244818 3988 tablet_bootstrap.cc:492] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d: Bootstrap starting.
I20260809 19:58:12.245745 3985 tablet_bootstrap.cc:654] T 5814361c2f24496fa296e86169236f87 P 58b4bccc13254c8988cf60c89783f89a: Neither blocks nor log segments found. Creating new log.
I20260809 19:58:12.246459 3985 log.cc:826] T 5814361c2f24496fa296e86169236f87 P 58b4bccc13254c8988cf60c89783f89a: Log is configured to *not* fsync() on all Append() calls
I20260809 19:58:12.246740 3988 tablet_bootstrap.cc:654] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d: Neither blocks nor log segments found. Creating new log.
I20260809 19:58:12.246740 3983 tablet_bootstrap.cc:654] T 5814361c2f24496fa296e86169236f87 P badc9e1d2ce547869d5d9ebf74a45b74: Neither blocks nor log segments found. Creating new log.
I20260809 19:58:12.247601 3988 log.cc:826] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d: Log is configured to *not* fsync() on all Append() calls
I20260809 19:58:12.247601 3983 log.cc:826] T 5814361c2f24496fa296e86169236f87 P badc9e1d2ce547869d5d9ebf74a45b74: Log is configured to *not* fsync() on all Append() calls
I20260809 19:58:12.248911 3985 tablet_bootstrap.cc:492] T 5814361c2f24496fa296e86169236f87 P 58b4bccc13254c8988cf60c89783f89a: No bootstrap required, opened a new log
I20260809 19:58:12.249120 3985 ts_tablet_manager.cc:1403] T 5814361c2f24496fa296e86169236f87 P 58b4bccc13254c8988cf60c89783f89a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 19:58:12.249573 3983 tablet_bootstrap.cc:492] T 5814361c2f24496fa296e86169236f87 P badc9e1d2ce547869d5d9ebf74a45b74: No bootstrap required, opened a new log
I20260809 19:58:12.249584 3988 tablet_bootstrap.cc:492] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d: No bootstrap required, opened a new log
I20260809 19:58:12.249761 3983 ts_tablet_manager.cc:1403] T 5814361c2f24496fa296e86169236f87 P badc9e1d2ce547869d5d9ebf74a45b74: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260809 19:58:12.249761 3988 ts_tablet_manager.cc:1403] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 19:58:12.251752 3985 raft_consensus.cc:359] T 5814361c2f24496fa296e86169236f87 P 58b4bccc13254c8988cf60c89783f89a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } } peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } }
I20260809 19:58:12.251946 3985 raft_consensus.cc:385] T 5814361c2f24496fa296e86169236f87 P 58b4bccc13254c8988cf60c89783f89a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 19:58:12.252019 3985 raft_consensus.cc:740] T 5814361c2f24496fa296e86169236f87 P 58b4bccc13254c8988cf60c89783f89a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58b4bccc13254c8988cf60c89783f89a, State: Initialized, Role: FOLLOWER
I20260809 19:58:12.252440 3985 consensus_queue.cc:260] T 5814361c2f24496fa296e86169236f87 P 58b4bccc13254c8988cf60c89783f89a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } } peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } }
I20260809 19:58:12.252908 3988 raft_consensus.cc:359] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } } peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } }
I20260809 19:58:12.253086 3602 heartbeater.cc:499] Master 127.3.56.62:33635 was elected leader, sending a full tablet report...
I20260809 19:58:12.253100 3988 raft_consensus.cc:385] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 19:58:12.253161 3988 raft_consensus.cc:740] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20da41f8404d40e59d12b8d87f881a1d, State: Initialized, Role: FOLLOWER
I20260809 19:58:12.253295 3985 ts_tablet_manager.cc:1434] T 5814361c2f24496fa296e86169236f87 P 58b4bccc13254c8988cf60c89783f89a: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260809 19:58:12.253571 3988 consensus_queue.cc:260] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } } peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } }
I20260809 19:58:12.253808 3983 raft_consensus.cc:359] T 5814361c2f24496fa296e86169236f87 P badc9e1d2ce547869d5d9ebf74a45b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } } peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } }
I20260809 19:58:12.254060 3983 raft_consensus.cc:385] T 5814361c2f24496fa296e86169236f87 P badc9e1d2ce547869d5d9ebf74a45b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 19:58:12.254158 3983 raft_consensus.cc:740] T 5814361c2f24496fa296e86169236f87 P badc9e1d2ce547869d5d9ebf74a45b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: badc9e1d2ce547869d5d9ebf74a45b74, State: Initialized, Role: FOLLOWER
I20260809 19:58:12.254184 3775 heartbeater.cc:499] Master 127.3.56.62:33635 was elected leader, sending a full tablet report...
I20260809 19:58:12.254936 3983 consensus_queue.cc:260] T 5814361c2f24496fa296e86169236f87 P badc9e1d2ce547869d5d9ebf74a45b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } } peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } }
I20260809 19:58:12.255570 3988 ts_tablet_manager.cc:1434] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 19:58:12.255708 3983 ts_tablet_manager.cc:1434] T 5814361c2f24496fa296e86169236f87 P badc9e1d2ce547869d5d9ebf74a45b74: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260809 19:58:12.255836 3988 tablet_bootstrap.cc:492] T 8375821116104cf999ae49469e06def0 P 20da41f8404d40e59d12b8d87f881a1d: Bootstrap starting.
I20260809 19:58:12.256016 3983 tablet_bootstrap.cc:492] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74: Bootstrap starting.
I20260809 19:58:12.256047 3960 heartbeater.cc:499] Master 127.3.56.62:33635 was elected leader, sending a full tablet report...
I20260809 19:58:12.257094 3983 tablet_bootstrap.cc:654] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74: Neither blocks nor log segments found. Creating new log.
I20260809 19:58:12.257187 3988 tablet_bootstrap.cc:654] T 8375821116104cf999ae49469e06def0 P 20da41f8404d40e59d12b8d87f881a1d: Neither blocks nor log segments found. Creating new log.
I20260809 19:58:12.258420 3985 tablet_bootstrap.cc:492] T 8375821116104cf999ae49469e06def0 P 58b4bccc13254c8988cf60c89783f89a: Bootstrap starting.
I20260809 19:58:12.258589 3983 tablet_bootstrap.cc:492] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74: No bootstrap required, opened a new log
I20260809 19:58:12.258674 3983 ts_tablet_manager.cc:1403] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 19:58:12.259223 3983 raft_consensus.cc:359] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } } peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } }
I20260809 19:58:12.259371 3983 raft_consensus.cc:385] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 19:58:12.259444 3983 raft_consensus.cc:740] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: badc9e1d2ce547869d5d9ebf74a45b74, State: Initialized, Role: FOLLOWER
I20260809 19:58:12.259591 3983 consensus_queue.cc:260] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } } peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } }
I20260809 19:58:12.259900 3983 ts_tablet_manager.cc:1434] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 19:58:12.260107 3983 tablet_bootstrap.cc:492] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74: Bootstrap starting.
I20260809 19:58:12.260188 3985 tablet_bootstrap.cc:654] T 8375821116104cf999ae49469e06def0 P 58b4bccc13254c8988cf60c89783f89a: Neither blocks nor log segments found. Creating new log.
I20260809 19:58:12.260767 3988 tablet_bootstrap.cc:492] T 8375821116104cf999ae49469e06def0 P 20da41f8404d40e59d12b8d87f881a1d: No bootstrap required, opened a new log
I20260809 19:58:12.260874 3988 ts_tablet_manager.cc:1403] T 8375821116104cf999ae49469e06def0 P 20da41f8404d40e59d12b8d87f881a1d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260809 19:58:12.261245 3983 tablet_bootstrap.cc:654] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74: Neither blocks nor log segments found. Creating new log.
I20260809 19:58:12.261453 3988 raft_consensus.cc:359] T 8375821116104cf999ae49469e06def0 P 20da41f8404d40e59d12b8d87f881a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } } peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } }
I20260809 19:58:12.261557 3988 raft_consensus.cc:385] T 8375821116104cf999ae49469e06def0 P 20da41f8404d40e59d12b8d87f881a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 19:58:12.261596 3988 raft_consensus.cc:740] T 8375821116104cf999ae49469e06def0 P 20da41f8404d40e59d12b8d87f881a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20da41f8404d40e59d12b8d87f881a1d, State: Initialized, Role: FOLLOWER
I20260809 19:58:12.261718 3988 consensus_queue.cc:260] T 8375821116104cf999ae49469e06def0 P 20da41f8404d40e59d12b8d87f881a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } } peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } }
I20260809 19:58:12.261873 3985 tablet_bootstrap.cc:492] T 8375821116104cf999ae49469e06def0 P 58b4bccc13254c8988cf60c89783f89a: No bootstrap required, opened a new log
I20260809 19:58:12.261948 3985 ts_tablet_manager.cc:1403] T 8375821116104cf999ae49469e06def0 P 58b4bccc13254c8988cf60c89783f89a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260809 19:58:12.261963 3988 ts_tablet_manager.cc:1434] T 8375821116104cf999ae49469e06def0 P 20da41f8404d40e59d12b8d87f881a1d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 19:58:12.262166 3988 tablet_bootstrap.cc:492] T 97d3cd24f4b24b578165968c3aa82513 P 20da41f8404d40e59d12b8d87f881a1d: Bootstrap starting.
I20260809 19:58:12.262475 3985 raft_consensus.cc:359] T 8375821116104cf999ae49469e06def0 P 58b4bccc13254c8988cf60c89783f89a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } } peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } }
I20260809 19:58:12.262588 3985 raft_consensus.cc:385] T 8375821116104cf999ae49469e06def0 P 58b4bccc13254c8988cf60c89783f89a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 19:58:12.262624 3985 raft_consensus.cc:740] T 8375821116104cf999ae49469e06def0 P 58b4bccc13254c8988cf60c89783f89a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58b4bccc13254c8988cf60c89783f89a, State: Initialized, Role: FOLLOWER
I20260809 19:58:12.262737 3985 consensus_queue.cc:260] T 8375821116104cf999ae49469e06def0 P 58b4bccc13254c8988cf60c89783f89a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } } peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } }
I20260809 19:58:12.262992 3985 ts_tablet_manager.cc:1434] T 8375821116104cf999ae49469e06def0 P 58b4bccc13254c8988cf60c89783f89a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 19:58:12.263185 3985 tablet_bootstrap.cc:492] T 97d3cd24f4b24b578165968c3aa82513 P 58b4bccc13254c8988cf60c89783f89a: Bootstrap starting.
I20260809 19:58:12.263207 3988 tablet_bootstrap.cc:654] T 97d3cd24f4b24b578165968c3aa82513 P 20da41f8404d40e59d12b8d87f881a1d: Neither blocks nor log segments found. Creating new log.
I20260809 19:58:12.264158 3985 tablet_bootstrap.cc:654] T 97d3cd24f4b24b578165968c3aa82513 P 58b4bccc13254c8988cf60c89783f89a: Neither blocks nor log segments found. Creating new log.
I20260809 19:58:12.264684 3983 tablet_bootstrap.cc:492] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74: No bootstrap required, opened a new log
I20260809 19:58:12.264770 3983 ts_tablet_manager.cc:1403] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260809 19:58:12.265301 3983 raft_consensus.cc:359] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } } peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } }
I20260809 19:58:12.265432 3983 raft_consensus.cc:385] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 19:58:12.265456 3985 tablet_bootstrap.cc:492] T 97d3cd24f4b24b578165968c3aa82513 P 58b4bccc13254c8988cf60c89783f89a: No bootstrap required, opened a new log
I20260809 19:58:12.265467 3983 raft_consensus.cc:740] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: badc9e1d2ce547869d5d9ebf74a45b74, State: Initialized, Role: FOLLOWER
I20260809 19:58:12.265542 3985 ts_tablet_manager.cc:1403] T 97d3cd24f4b24b578165968c3aa82513 P 58b4bccc13254c8988cf60c89783f89a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.001s
I20260809 19:58:12.265611 3983 consensus_queue.cc:260] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } } peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } }
I20260809 19:58:12.265913 3983 ts_tablet_manager.cc:1434] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 19:58:12.265959 3985 raft_consensus.cc:359] T 97d3cd24f4b24b578165968c3aa82513 P 58b4bccc13254c8988cf60c89783f89a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } } peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } }
I20260809 19:58:12.266036 3985 raft_consensus.cc:385] T 97d3cd24f4b24b578165968c3aa82513 P 58b4bccc13254c8988cf60c89783f89a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 19:58:12.266063 3985 raft_consensus.cc:740] T 97d3cd24f4b24b578165968c3aa82513 P 58b4bccc13254c8988cf60c89783f89a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58b4bccc13254c8988cf60c89783f89a, State: Initialized, Role: FOLLOWER
I20260809 19:58:12.266173 3985 consensus_queue.cc:260] T 97d3cd24f4b24b578165968c3aa82513 P 58b4bccc13254c8988cf60c89783f89a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } } peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } }
I20260809 19:58:12.266402 3985 ts_tablet_manager.cc:1434] T 97d3cd24f4b24b578165968c3aa82513 P 58b4bccc13254c8988cf60c89783f89a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 19:58:12.266433 3988 tablet_bootstrap.cc:492] T 97d3cd24f4b24b578165968c3aa82513 P 20da41f8404d40e59d12b8d87f881a1d: No bootstrap required, opened a new log
I20260809 19:58:12.266515 3988 ts_tablet_manager.cc:1403] T 97d3cd24f4b24b578165968c3aa82513 P 20da41f8404d40e59d12b8d87f881a1d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260809 19:58:12.266918 3995 raft_consensus.cc:493] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 19:58:12.267051 3995 raft_consensus.cc:515] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } } peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } }
I20260809 19:58:12.267110 3988 raft_consensus.cc:359] T 97d3cd24f4b24b578165968c3aa82513 P 20da41f8404d40e59d12b8d87f881a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } } peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } }
I20260809 19:58:12.267202 3988 raft_consensus.cc:385] T 97d3cd24f4b24b578165968c3aa82513 P 20da41f8404d40e59d12b8d87f881a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 19:58:12.267233 3988 raft_consensus.cc:740] T 97d3cd24f4b24b578165968c3aa82513 P 20da41f8404d40e59d12b8d87f881a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20da41f8404d40e59d12b8d87f881a1d, State: Initialized, Role: FOLLOWER
I20260809 19:58:12.267364 3988 consensus_queue.cc:260] T 97d3cd24f4b24b578165968c3aa82513 P 20da41f8404d40e59d12b8d87f881a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } } peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } }
I20260809 19:58:12.267732 3988 ts_tablet_manager.cc:1434] T 97d3cd24f4b24b578165968c3aa82513 P 20da41f8404d40e59d12b8d87f881a1d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 19:58:12.268225 3995 leader_election.cc:290] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591), badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687)
I20260809 19:58:12.272596 3894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5814361c2f24496fa296e86169236f87" candidate_uuid: "20da41f8404d40e59d12b8d87f881a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" is_pre_election: true
I20260809 19:58:12.272652 3543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5814361c2f24496fa296e86169236f87" candidate_uuid: "20da41f8404d40e59d12b8d87f881a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58b4bccc13254c8988cf60c89783f89a" is_pre_election: true
I20260809 19:58:12.272814 3894 raft_consensus.cc:2468] T 5814361c2f24496fa296e86169236f87 P badc9e1d2ce547869d5d9ebf74a45b74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20da41f8404d40e59d12b8d87f881a1d in term 0.
I20260809 19:58:12.272907 3543 raft_consensus.cc:2468] T 5814361c2f24496fa296e86169236f87 P 58b4bccc13254c8988cf60c89783f89a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20da41f8404d40e59d12b8d87f881a1d in term 0.
W20260809 19:58:12.273216 3776 tablet.cc:2406] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 19:58:12.273298 3633 leader_election.cc:304] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20da41f8404d40e59d12b8d87f881a1d, badc9e1d2ce547869d5d9ebf74a45b74; no voters:
I20260809 19:58:12.273604 4008 raft_consensus.cc:2804] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 19:58:12.273675 4008 raft_consensus.cc:493] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 19:58:12.273726 4008 raft_consensus.cc:3060] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d [term 0 FOLLOWER]: Advancing to term 1
I20260809 19:58:12.274667 4008 raft_consensus.cc:515] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } } peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } }
I20260809 19:58:12.275002 4008 leader_election.cc:290] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d [CANDIDATE]: Term 1 election: Requested vote from peers 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591), badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687)
I20260809 19:58:12.275305 3543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5814361c2f24496fa296e86169236f87" candidate_uuid: "20da41f8404d40e59d12b8d87f881a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58b4bccc13254c8988cf60c89783f89a"
I20260809 19:58:12.275411 3894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5814361c2f24496fa296e86169236f87" candidate_uuid: "20da41f8404d40e59d12b8d87f881a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "badc9e1d2ce547869d5d9ebf74a45b74"
I20260809 19:58:12.275422 3543 raft_consensus.cc:3060] T 5814361c2f24496fa296e86169236f87 P 58b4bccc13254c8988cf60c89783f89a [term 0 FOLLOWER]: Advancing to term 1
I20260809 19:58:12.275519 3894 raft_consensus.cc:3060] T 5814361c2f24496fa296e86169236f87 P badc9e1d2ce547869d5d9ebf74a45b74 [term 0 FOLLOWER]: Advancing to term 1
I20260809 19:58:12.276182 3997 raft_consensus.cc:493] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 19:58:12.276295 3997 raft_consensus.cc:515] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } } peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } }
I20260809 19:58:12.276753 3894 raft_consensus.cc:2468] T 5814361c2f24496fa296e86169236f87 P badc9e1d2ce547869d5d9ebf74a45b74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20da41f8404d40e59d12b8d87f881a1d in term 1.
I20260809 19:58:12.276783 3543 raft_consensus.cc:2468] T 5814361c2f24496fa296e86169236f87 P 58b4bccc13254c8988cf60c89783f89a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20da41f8404d40e59d12b8d87f881a1d in term 1.
I20260809 19:58:12.277024 3633 leader_election.cc:304] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20da41f8404d40e59d12b8d87f881a1d, badc9e1d2ce547869d5d9ebf74a45b74; no voters:
I20260809 19:58:12.277326 3997 leader_election.cc:290] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591), 20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843)
I20260809 19:58:12.277496 4008 raft_consensus.cc:2804] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 19:58:12.277637 4008 raft_consensus.cc:697] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d [term 1 LEADER]: Becoming Leader. State: Replica: 20da41f8404d40e59d12b8d87f881a1d, State: Running, Role: LEADER
I20260809 19:58:12.277925 4008 consensus_queue.cc:237] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } } peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } }
I20260809 19:58:12.281172 3345 catalog_manager.cc:5719] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 20da41f8404d40e59d12b8d87f881a1d (127.3.56.2). New cstate: current_term: 1 leader_uuid: "20da41f8404d40e59d12b8d87f881a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } health_report { overall_health: UNKNOWN } } }
I20260809 19:58:12.281450 3718 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8375821116104cf999ae49469e06def0" candidate_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20da41f8404d40e59d12b8d87f881a1d" is_pre_election: true
I20260809 19:58:12.281639 3718 raft_consensus.cc:2468] T 8375821116104cf999ae49469e06def0 P 20da41f8404d40e59d12b8d87f881a1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate badc9e1d2ce547869d5d9ebf74a45b74 in term 0.
I20260809 19:58:12.282023 3802 leader_election.cc:304] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20da41f8404d40e59d12b8d87f881a1d, badc9e1d2ce547869d5d9ebf74a45b74; no voters:
I20260809 19:58:12.282239 3997 raft_consensus.cc:2804] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 19:58:12.282305 3997 raft_consensus.cc:493] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 19:58:12.282358 3997 raft_consensus.cc:3060] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 [term 0 FOLLOWER]: Advancing to term 1
I20260809 19:58:12.283303 3997 raft_consensus.cc:515] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } } peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } }
I20260809 19:58:12.283639 3543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8375821116104cf999ae49469e06def0" candidate_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58b4bccc13254c8988cf60c89783f89a" is_pre_election: true
I20260809 19:58:12.283679 3997 leader_election.cc:290] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 [CANDIDATE]: Term 1 election: Requested vote from peers 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591), 20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843)
I20260809 19:58:12.283736 3543 raft_consensus.cc:2468] T 8375821116104cf999ae49469e06def0 P 58b4bccc13254c8988cf60c89783f89a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate badc9e1d2ce547869d5d9ebf74a45b74 in term 0.
I20260809 19:58:12.284241 3543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8375821116104cf999ae49469e06def0" candidate_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58b4bccc13254c8988cf60c89783f89a"
I20260809 19:58:12.284235 3718 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8375821116104cf999ae49469e06def0" candidate_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20da41f8404d40e59d12b8d87f881a1d"
I20260809 19:58:12.284348 3718 raft_consensus.cc:3060] T 8375821116104cf999ae49469e06def0 P 20da41f8404d40e59d12b8d87f881a1d [term 0 FOLLOWER]: Advancing to term 1
I20260809 19:58:12.284346 3543 raft_consensus.cc:3060] T 8375821116104cf999ae49469e06def0 P 58b4bccc13254c8988cf60c89783f89a [term 0 FOLLOWER]: Advancing to term 1
I20260809 19:58:12.285243 3718 raft_consensus.cc:2468] T 8375821116104cf999ae49469e06def0 P 20da41f8404d40e59d12b8d87f881a1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate badc9e1d2ce547869d5d9ebf74a45b74 in term 1.
I20260809 19:58:12.285243 3543 raft_consensus.cc:2468] T 8375821116104cf999ae49469e06def0 P 58b4bccc13254c8988cf60c89783f89a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate badc9e1d2ce547869d5d9ebf74a45b74 in term 1.
I20260809 19:58:12.285564 3802 leader_election.cc:304] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58b4bccc13254c8988cf60c89783f89a, badc9e1d2ce547869d5d9ebf74a45b74; no voters:
I20260809 19:58:12.285720 3997 raft_consensus.cc:2804] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 19:58:12.285925 3997 raft_consensus.cc:697] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 [term 1 LEADER]: Becoming Leader. State: Replica: badc9e1d2ce547869d5d9ebf74a45b74, State: Running, Role: LEADER
I20260809 19:58:12.286238 3997 consensus_queue.cc:237] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } } peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } }
I20260809 19:58:12.289193 3345 catalog_manager.cc:5719] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 reported cstate change: term changed from 0 to 1, leader changed from <none> to badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3). New cstate: current_term: 1 leader_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } health_report { overall_health: HEALTHY } } }
W20260809 19:58:12.344842 3604 tablet.cc:2406] T 5814361c2f24496fa296e86169236f87 P 58b4bccc13254c8988cf60c89783f89a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 19:58:12.440568 3961 tablet.cc:2406] T 5814361c2f24496fa296e86169236f87 P badc9e1d2ce547869d5d9ebf74a45b74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 19:58:12.476613 3997 raft_consensus.cc:493] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 19:58:12.476749 3997 raft_consensus.cc:515] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } } peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } }
I20260809 19:58:12.477211 3997 leader_election.cc:290] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843), 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591)
I20260809 19:58:12.477865 3718 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97d3cd24f4b24b578165968c3aa82513" candidate_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20da41f8404d40e59d12b8d87f881a1d" is_pre_election: true
I20260809 19:58:12.477866 3543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97d3cd24f4b24b578165968c3aa82513" candidate_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58b4bccc13254c8988cf60c89783f89a" is_pre_election: true
I20260809 19:58:12.478027 3543 raft_consensus.cc:2468] T 97d3cd24f4b24b578165968c3aa82513 P 58b4bccc13254c8988cf60c89783f89a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate badc9e1d2ce547869d5d9ebf74a45b74 in term 0.
I20260809 19:58:12.478027 3718 raft_consensus.cc:2468] T 97d3cd24f4b24b578165968c3aa82513 P 20da41f8404d40e59d12b8d87f881a1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate badc9e1d2ce547869d5d9ebf74a45b74 in term 0.
I20260809 19:58:12.478380 3802 leader_election.cc:304] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58b4bccc13254c8988cf60c89783f89a, badc9e1d2ce547869d5d9ebf74a45b74; no voters:
I20260809 19:58:12.478569 3997 raft_consensus.cc:2804] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 19:58:12.478631 3997 raft_consensus.cc:493] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 19:58:12.478693 3997 raft_consensus.cc:3060] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 [term 0 FOLLOWER]: Advancing to term 1
I20260809 19:58:12.479499 3997 raft_consensus.cc:515] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } } peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } }
I20260809 19:58:12.479830 3997 leader_election.cc:290] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 [CANDIDATE]: Term 1 election: Requested vote from peers 20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843), 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591)
I20260809 19:58:12.480230 3718 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97d3cd24f4b24b578165968c3aa82513" candidate_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20da41f8404d40e59d12b8d87f881a1d"
I20260809 19:58:12.480252 3543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97d3cd24f4b24b578165968c3aa82513" candidate_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58b4bccc13254c8988cf60c89783f89a"
I20260809 19:58:12.480381 3718 raft_consensus.cc:3060] T 97d3cd24f4b24b578165968c3aa82513 P 20da41f8404d40e59d12b8d87f881a1d [term 0 FOLLOWER]: Advancing to term 1
I20260809 19:58:12.480379 3543 raft_consensus.cc:3060] T 97d3cd24f4b24b578165968c3aa82513 P 58b4bccc13254c8988cf60c89783f89a [term 0 FOLLOWER]: Advancing to term 1
I20260809 19:58:12.481375 3543 raft_consensus.cc:2468] T 97d3cd24f4b24b578165968c3aa82513 P 58b4bccc13254c8988cf60c89783f89a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate badc9e1d2ce547869d5d9ebf74a45b74 in term 1.
I20260809 19:58:12.481372 3718 raft_consensus.cc:2468] T 97d3cd24f4b24b578165968c3aa82513 P 20da41f8404d40e59d12b8d87f881a1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate badc9e1d2ce547869d5d9ebf74a45b74 in term 1.
I20260809 19:58:12.481680 3802 leader_election.cc:304] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20da41f8404d40e59d12b8d87f881a1d, badc9e1d2ce547869d5d9ebf74a45b74; no voters:
I20260809 19:58:12.481843 3997 raft_consensus.cc:2804] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 19:58:12.481931 3997 raft_consensus.cc:697] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 [term 1 LEADER]: Becoming Leader. State: Replica: badc9e1d2ce547869d5d9ebf74a45b74, State: Running, Role: LEADER
I20260809 19:58:12.482081 3997 consensus_queue.cc:237] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } } peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } }
I20260809 19:58:12.483767 3345 catalog_manager.cc:5719] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 reported cstate change: term changed from 0 to 1, leader changed from <none> to badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3). New cstate: current_term: 1 leader_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 } health_report { overall_health: UNKNOWN } } }
I20260809 19:58:12.664794 4008 consensus_queue.cc:1048] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "badc9e1d2ce547869d5d9ebf74a45b74" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 46687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 19:58:12.669656 4008 consensus_queue.cc:1048] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 19:58:12.712983 3997 consensus_queue.cc:1048] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 19:58:12.716867 4013 consensus_queue.cc:1048] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 19:58:12.982549 3997 consensus_queue.cc:1048] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58b4bccc13254c8988cf60c89783f89a" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 37591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 19:58:12.986783 3997 consensus_queue.cc:1048] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20da41f8404d40e59d12b8d87f881a1d" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 34843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 19:58:14.194008 3961 maintenance_manager.cc:419] P badc9e1d2ce547869d5d9ebf74a45b74: Scheduling FlushMRSOp(97d3cd24f4b24b578165968c3aa82513): perf score=1.000000
I20260809 19:58:14.348927 3604 maintenance_manager.cc:419] P 58b4bccc13254c8988cf60c89783f89a: Scheduling FlushMRSOp(8375821116104cf999ae49469e06def0): perf score=1.000000
W20260809 19:58:14.493268 3969 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.3.56.3:46687, user_credentials={real_user=slave}} blocked reactor thread for 106707us
I20260809 19:58:14.527619 3776 maintenance_manager.cc:419] P 20da41f8404d40e59d12b8d87f881a1d: Scheduling FlushMRSOp(8375821116104cf999ae49469e06def0): perf score=1.937278
W20260809 19:58:14.716120 3969 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.3.56.3:46687, user_credentials={real_user=slave}} blocked reactor thread for 218553us
W20260809 19:58:15.116500 3458 dir_manager.cc:275] Insufficient disk space under path /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-0/data/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-0/data/data (493703168 bytes available vs 518351011 bytes reserved) (error 28)
W20260809 19:58:15.122936 3458 tablet_replica_mm_ops.cc:214] T 8375821116104cf999ae49469e06def0 P 58b4bccc13254c8988cf60c89783f89a: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: Couldn't allocate a block for bloom filter: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 8375821116104cf999ae49469e06def0's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
F20260809 19:58:15.123160 3458 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: Couldn't allocate a block for bloom filter: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 8375821116104cf999ae49469e06def0's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
*** Check failure stack trace: ***
@ 0x7f380f060d5d google::LogMessage::Fail() at ??:0
@ 0x7f380f064b23 google::LogMessage::SendToLog() at ??:0
@ 0x7f380f06075c google::LogMessage::Flush() at ??:0
@ 0x7f380f061ee9 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3812aeab22 kudu::tablet::FlushMRSOp::Perform() at ??:0
W20260809 19:58:15.154297 3963 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.3.56.2:34843, user_credentials={real_user=slave}} blocked reactor thread for 86596.1us
@ 0x7f3810583cf2 kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7f3810580e08 _ZZN4kudu18MaintenanceManager18RunSchedulerThreadEvENKUlvE_clEv at ??:0
@ 0x7f3810585139 _ZNSt17_Function_handlerIFvvEZN4kudu18MaintenanceManager18RunSchedulerThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5591569cdb9c std::function<>::operator()() at ??:0
@ 0x7f3810541767 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f381054208f _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f38105437db _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5591569cdb9c std::function<>::operator()() at ??:0
@ 0x7f3810531b52 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3810eaf6db start_thread at ??:0
@ 0x7f380e21761f clone at ??:0
I20260809 19:58:15.247761 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:58:15.252393 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
W20260809 19:58:15.463861 3802 connection.cc:573] client connection to 127.3.56.1:37591 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 19:58:15.464054 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 19:58:15.465545 4119 negotiation.cc:336] Failed RPC negotiation. Trace:
0809 19:58:15.226691 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.3.56.1:37591 (local address 127.0.0.1:47244)
0809 19:58:15.231572 (+ 4881us) negotiation.cc:107] Waiting for socket to connect
0809 19:58:15.231586 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0809 19:58:15.232078 (+ 492us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0809 19:58:15.464942 (+232864us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":4811,"thread_start_us":67,"threads_started":1}
W20260809 19:58:15.465945 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:15.474406 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:15.546360 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111)
W20260809 19:58:15.547284 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 19:58:15.608181 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 19:58:15.669888 3634 maintenance_manager.cc:643] P 20da41f8404d40e59d12b8d87f881a1d: FlushMRSOp(8375821116104cf999ae49469e06def0) complete. Timing: real 1.142s user 0.623s sys 0.071s Metrics: {"bytes_written":473415,"cfile_init":1,"compiler_manager_pool.queue_time_us":23950,"compiler_manager_pool.run_cpu_time_us":333591,"compiler_manager_pool.run_wall_time_us":591072,"dirs.queue_time_us":196,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":758,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2738,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":283,"threads_started":3,"wal-append.queue_time_us":204}
I20260809 19:58:15.671331 3776 maintenance_manager.cc:419] P 20da41f8404d40e59d12b8d87f881a1d: Scheduling FlushMRSOp(5814361c2f24496fa296e86169236f87): perf score=1.937278
I20260809 19:58:15.678544 3806 maintenance_manager.cc:643] P badc9e1d2ce547869d5d9ebf74a45b74: FlushMRSOp(97d3cd24f4b24b578165968c3aa82513) complete. Timing: real 1.484s user 0.643s sys 0.031s Metrics: {"bytes_written":437013,"cfile_init":1,"compiler_manager_pool.queue_time_us":12190,"compiler_manager_pool.run_cpu_time_us":323734,"compiler_manager_pool.run_wall_time_us":420903,"dirs.queue_time_us":191,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":966,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2637,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"spinlock_wait_cycles":211328,"thread_start_us":313,"threads_started":3,"wal-append.queue_time_us":163}
I20260809 19:58:15.679881 3961 maintenance_manager.cc:419] P badc9e1d2ce547869d5d9ebf74a45b74: Scheduling FlushMRSOp(5814361c2f24496fa296e86169236f87): perf score=1.937278
I20260809 19:58:15.689802 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:15.692883 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:15.719527 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 19:58:16.010509 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:16.011550 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:16.049953 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 19:58:16.066293 3634 maintenance_manager.cc:643] P 20da41f8404d40e59d12b8d87f881a1d: FlushMRSOp(5814361c2f24496fa296e86169236f87) complete. Timing: real 0.395s user 0.390s sys 0.001s Metrics: {"bytes_written":474605,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1337,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2765,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000}
I20260809 19:58:16.066902 3776 maintenance_manager.cc:419] P 20da41f8404d40e59d12b8d87f881a1d: Scheduling UndoDeltaBlockGCOp(5814361c2f24496fa296e86169236f87): 63430 bytes on disk
I20260809 19:58:16.067685 3634 maintenance_manager.cc:643] P 20da41f8404d40e59d12b8d87f881a1d: UndoDeltaBlockGCOp(5814361c2f24496fa296e86169236f87) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20260809 19:58:16.068169 3776 maintenance_manager.cc:419] P 20da41f8404d40e59d12b8d87f881a1d: Scheduling UndoDeltaBlockGCOp(8375821116104cf999ae49469e06def0): 62920 bytes on disk
I20260809 19:58:16.068523 3634 maintenance_manager.cc:643] P 20da41f8404d40e59d12b8d87f881a1d: UndoDeltaBlockGCOp(8375821116104cf999ae49469e06def0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20260809 19:58:16.068921 3776 maintenance_manager.cc:419] P 20da41f8404d40e59d12b8d87f881a1d: Scheduling FlushMRSOp(97d3cd24f4b24b578165968c3aa82513): perf score=1.937278
I20260809 19:58:16.083032 3806 maintenance_manager.cc:643] P badc9e1d2ce547869d5d9ebf74a45b74: FlushMRSOp(5814361c2f24496fa296e86169236f87) complete. Timing: real 0.403s user 0.381s sys 0.012s Metrics: {"bytes_written":474605,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":252,"dirs.run_wall_time_us":1277,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2857,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000}
I20260809 19:58:16.083653 3961 maintenance_manager.cc:419] P badc9e1d2ce547869d5d9ebf74a45b74: Scheduling FlushMRSOp(8375821116104cf999ae49469e06def0): perf score=1.937278
I20260809 19:58:16.395704 3634 maintenance_manager.cc:643] P 20da41f8404d40e59d12b8d87f881a1d: FlushMRSOp(97d3cd24f4b24b578165968c3aa82513) complete. Timing: real 0.327s user 0.323s sys 0.001s Metrics: {"bytes_written":437013,"cfile_init":1,"dirs.queue_time_us":69,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1313,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2226,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":94,"threads_started":1,"wal-append.queue_time_us":184}
I20260809 19:58:16.396389 3776 maintenance_manager.cc:419] P 20da41f8404d40e59d12b8d87f881a1d: Scheduling UndoDeltaBlockGCOp(97d3cd24f4b24b578165968c3aa82513): 62920 bytes on disk
I20260809 19:58:16.396880 3634 maintenance_manager.cc:643] P 20da41f8404d40e59d12b8d87f881a1d: UndoDeltaBlockGCOp(97d3cd24f4b24b578165968c3aa82513) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20260809 19:58:16.409214 3806 maintenance_manager.cc:643] P badc9e1d2ce547869d5d9ebf74a45b74: FlushMRSOp(8375821116104cf999ae49469e06def0) complete. Timing: real 0.325s user 0.323s sys 0.000s Metrics: {"bytes_written":473415,"cfile_init":1,"dirs.queue_time_us":62,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1244,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2254,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":146,"threads_started":1,"wal-append.queue_time_us":230}
I20260809 19:58:16.409879 3961 maintenance_manager.cc:419] P badc9e1d2ce547869d5d9ebf74a45b74: Scheduling UndoDeltaBlockGCOp(5814361c2f24496fa296e86169236f87): 63430 bytes on disk
I20260809 19:58:16.410466 3806 maintenance_manager.cc:643] P badc9e1d2ce547869d5d9ebf74a45b74: UndoDeltaBlockGCOp(5814361c2f24496fa296e86169236f87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20260809 19:58:16.410864 3961 maintenance_manager.cc:419] P badc9e1d2ce547869d5d9ebf74a45b74: Scheduling UndoDeltaBlockGCOp(8375821116104cf999ae49469e06def0): 62920 bytes on disk
I20260809 19:58:16.411192 3806 maintenance_manager.cc:643] P badc9e1d2ce547869d5d9ebf74a45b74: UndoDeltaBlockGCOp(8375821116104cf999ae49469e06def0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20260809 19:58:16.411535 3961 maintenance_manager.cc:419] P badc9e1d2ce547869d5d9ebf74a45b74: Scheduling UndoDeltaBlockGCOp(97d3cd24f4b24b578165968c3aa82513): 62920 bytes on disk
I20260809 19:58:16.411826 3806 maintenance_manager.cc:643] P badc9e1d2ce547869d5d9ebf74a45b74: UndoDeltaBlockGCOp(97d3cd24f4b24b578165968c3aa82513) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4}
I20260809 19:58:16.442958 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:16.443034 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:16.466419 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 19:58:17.023478 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:17.023972 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:17.048357 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 19:58:17.857765 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:58:17.857861 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:17.882717 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:17.994733 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 19:58:18.012007 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 19:58:18.031678 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260809 19:58:18.931454 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:58:18.931737 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:18.951258 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 19:58:20.001773 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:20.002617 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:20.023681 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:20.512755 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 19:58:20.513597 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 19:58:20.565551 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 19:58:20.571949 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 19:58:20.572741 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260809 19:58:21.073818 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:58:21.073885 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:21.096292 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 19:58:22.147812 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:58:22.147804 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:22.167994 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:22.667708 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 19:58:22.986079 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 19:58:23.117105 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260809 19:58:23.218204 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:23.218523 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:23.238570 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 19:58:24.290830 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:58:24.291179 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:24.316813 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:25.279498 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260809 19:58:25.367597 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:25.367659 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
W20260809 19:58:25.388705 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260809 19:58:25.388998 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:25.526338 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 19:58:25.696870 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 19:58:25.926486 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20260809 19:58:26.438169 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:58:26.438212 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:26.457219 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:27.409250 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260809 19:58:27.510993 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:58:27.511066 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:27.535959 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:27.809067 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 19:58:27.913869 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260809 19:58:28.586221 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:58:28.586221 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:28.610011 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 19:58:29.659148 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:29.660226 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:29.681466 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:29.891963 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 19:58:30.111949 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260809 19:58:30.732967 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:30.733506 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
W20260809 19:58:30.738350 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:30.753144 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:31.063120 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 19:58:31.148543 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260809 19:58:31.803365 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:31.804723 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:31.827598 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:32.333318 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 19:58:32.714706 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260809 19:58:32.880419 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:32.880419 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:32.904129 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:33.000686 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260809 19:58:33.952780 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:33.953006 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:33.972889 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:34.819397 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260809 19:58:35.023334 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:35.023342 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
W20260809 19:58:35.037717 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:35.050899 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:35.354249 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260809 19:58:36.100003 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:36.100286 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:36.120440 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:36.321425 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 19:58:36.336969 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20260809 19:58:37.170904 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:58:37.170957 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:37.191156 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:37.289415 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 19:58:37.500422 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 19:58:37.827890 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260809 19:58:38.246438 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:38.247345 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:38.266847 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 19:58:39.317943 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:39.318887 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:39.340464 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:39.880004 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 19:58:39.949517 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260809 19:58:40.389413 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:40.389520 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:40.410182 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:40.602982 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 19:58:41.375065 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20260809 19:58:41.461102 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:58:41.461711 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:41.481845 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:41.580721 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 19:58:42.235509 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 19:58:42.474308 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260809 19:58:42.531595 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:42.531698 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:42.554817 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:43.138118 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260809 19:58:43.609202 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:43.609222 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:43.627658 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 19:58:44.676633 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:58:44.677052 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:44.701670 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:44.945830 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 19:58:45.172731 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260809 19:58:45.751637 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:58:45.751731 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:45.773073 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:45.774175 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 19:58:46.484936 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260809 19:58:46.693606 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260809 19:58:46.824463 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:46.824481 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:46.844694 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:47.553835 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260809 19:58:47.722775 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260809 19:58:47.896894 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:47.897132 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:47.920454 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:47.946187 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260809 19:58:48.977030 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:48.977195 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:48.997720 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 19:58:50.052618 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:50.053208 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:50.077591 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:50.078785 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260809 19:58:50.165450 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260809 19:58:50.516980 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260809 19:58:51.129635 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:58:51.130015 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:51.154567 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:51.617995 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 19:58:51.808812 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20260809 19:58:52.207373 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:52.207927 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:52.227236 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:52.649549 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260809 19:58:52.888401 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260809 19:58:52.946317 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20260809 19:58:53.276813 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:58:53.277071 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:53.300344 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 19:58:54.350199 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:54.350557 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:54.371820 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:54.904348 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20260809 19:58:55.423218 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:55.423659 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
W20260809 19:58:55.434295 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:55.446293 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:55.691286 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20260809 19:58:56.497689 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:58:56.497726 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:56.517959 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:57.000104 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 19:58:57.321861 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 19:58:57.438658 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20260809 19:58:57.568992 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:58:57.569355 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:57.595149 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:58:58.057294 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20260809 19:58:58.466411 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20260809 19:58:58.645553 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:58:58.645612 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:58.669965 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 19:58:59.724448 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:58:59.724679 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:58:59.749699 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:00.046602 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20260809 19:59:00.698298 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20260809 19:59:00.802868 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:00.802975 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:00.825155 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:01.118968 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20260809 19:59:01.890203 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:01.890549 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:01.923869 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:02.069947 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 19:59:02.561487 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 19:59:02.646859 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20260809 19:59:02.976171 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:59:02.976171 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:03.002192 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:03.358127 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20260809 19:59:03.543679 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20260809 19:59:04.055244 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:59:04.055289 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:04.076376 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 19:59:05.132266 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:05.132266 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:05.155591 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:05.284765 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20260809 19:59:05.995621 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20260809 19:59:06.064039 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20260809 19:59:06.219254 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:06.219326 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:06.245965 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:07.221516 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20260809 19:59:07.297883 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:07.298285 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:07.317168 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:07.714011 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 19:59:07.716581 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20260809 19:59:08.261263 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20260809 19:59:08.367756 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:59:08.367777 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:08.393550 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:08.685895 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20260809 19:59:09.446573 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:59:09.446557 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:09.466846 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:10.492481 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20260809 19:59:10.521328 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:10.521409 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:10.541697 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:10.987484 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20260809 19:59:11.181106 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20260809 19:59:11.594058 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:11.594053 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:11.623670 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:12.301317 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20260809 19:59:12.675383 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:12.675644 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:12.695952 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:12.815258 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 19:59:12.979229 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20260809 19:59:13.286339 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20260809 19:59:13.748353 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:13.748353 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:13.772527 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:13.830674 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20260809 19:59:14.824901 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:14.824995 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:14.848135 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:15.288448 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20260809 19:59:15.842381 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20260809 19:59:15.902788 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:59:15.902992 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:15.923758 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:16.286360 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20260809 19:59:16.974668 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:59:16.974668 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:16.998947 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:17.602658 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20260809 19:59:18.052194 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:59:18.052393 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:18.077020 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:18.184180 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20260809 19:59:18.187338 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 19:59:18.188023 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20260809 19:59:18.719868 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20260809 19:59:19.132910 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:19.133944 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:19.154313 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 19:59:20.205075 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:20.205088 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:20.231047 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:20.790186 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20260809 19:59:20.810356 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20260809 19:59:21.085665 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20260809 19:59:21.285012 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:59:21.285770 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:21.304601 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 19:59:22.356036 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:59:22.356141 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:22.375324 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:22.614107 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 19:59:23.186004 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20260809 19:59:23.314759 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 19:59:23.315675 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20260809 19:59:23.426610 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:23.426899 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:23.447243 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:23.506806 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20260809 19:59:24.497952 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:59:24.498126 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:24.518674 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 19:59:25.570030 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:25.570125 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:25.592813 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:25.703590 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20260809 19:59:25.765985 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20260809 19:59:25.952524 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20260809 19:59:26.646827 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:26.646850 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:26.667721 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 19:59:27.718571 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:27.718720 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:27.743187 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:28.078768 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 19:59:28.085800 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20260809 19:59:28.089542 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20260809 19:59:28.541594 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20260809 19:59:28.673985 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20260809 19:59:28.792614 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:59:28.792701 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:28.814781 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 19:59:29.864764 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:29.864874 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:29.887655 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:30.702239 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20260809 19:59:30.808795 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20260809 19:59:30.938817 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:30.939149 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:30.960448 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:31.169924 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20260809 19:59:32.014038 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:32.014057 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:32.034214 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 19:59:33.086638 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:59:33.089254 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:33.110085 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:33.141717 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 19:59:33.146641 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20260809 19:59:33.477131 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20260809 19:59:33.730713 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20260809 19:59:34.050721 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20260809 19:59:34.161015 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:34.161048 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:34.181160 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 19:59:35.231917 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:59:35.231973 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:35.251462 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:35.427004 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20260809 19:59:36.054437 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20260809 19:59:36.303540 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:36.303853 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:36.323056 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:36.411567 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20260809 19:59:37.377889 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:37.377949 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:37.402020 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:37.994189 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20260809 19:59:38.283476 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 19:59:38.284246 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20260809 19:59:38.451886 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:38.452345 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:38.475584 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:39.028803 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20260809 19:59:39.061388 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260809 19:59:39.528026 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:39.528084 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:39.550307 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:40.551774 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20260809 19:59:40.603891 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:40.612632 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:40.632933 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:40.895358 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20260809 19:59:41.682610 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:41.682878 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:41.704526 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:41.770824 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20260809 19:59:42.762428 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:42.763186 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:42.783838 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:42.914279 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20260809 19:59:43.254038 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20260809 19:59:43.289240 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20260809 19:59:43.835722 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:59:43.836670 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:43.861168 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:44.066309 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 19:59:44.287540 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20260809 19:59:44.913619 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:59:44.913619 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:44.934695 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:45.422763 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20260809 19:59:45.614401 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20260809 19:59:45.985518 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:45.985739 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:46.005112 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:46.590189 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20260809 19:59:47.055354 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:47.055652 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:47.075891 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:47.799844 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20260809 19:59:47.915536 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20260809 19:59:48.130479 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:59:48.130934 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:48.153913 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:48.305836 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20260809 19:59:49.204237 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:59:49.207594 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:49.226074 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:49.254612 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20260809 19:59:49.362644 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 19:59:50.181016 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20260809 19:59:50.259972 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20260809 19:59:50.278357 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:59:50.278357 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:50.305814 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 19:59:51.358353 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:59:51.358596 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:51.380085 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:51.984830 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20260809 19:59:52.440668 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:52.440685 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:52.465988 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:52.662138 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20260809 19:59:52.778108 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20260809 19:59:53.333113 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20260809 19:59:53.524071 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:59:53.524665 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:53.545423 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:54.365728 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20260809 19:59:54.595490 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:54.595543 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:54.618408 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:54.743813 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 19:59:55.154734 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20260809 19:59:55.246393 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20260809 19:59:55.669155 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:59:55.669754 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:55.689833 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:56.687309 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20260809 19:59:56.739675 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:59:56.739880 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:56.763393 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:57.460987 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20260809 19:59:57.614601 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20260809 19:59:57.813547 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 19:59:57.813882 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:57.834295 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:58.520931 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20260809 19:59:58.889142 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:59:58.889779 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:58.910422 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 19:59:59.102089 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20260809 19:59:59.871199 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 19:59:59.871949 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20260809 19:59:59.961858 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 19:59:59.962296 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 19:59:59.984918 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 20:00:00.135601 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20260809 20:00:01.034720 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 20:00:01.034783 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 20:00:01.057871 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 20:00:02.013955 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20260809 20:00:02.154672 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 20:00:02.154832 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 20:00:02.177482 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 20:00:02.527033 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20260809 20:00:02.747669 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20260809 20:00:03.228915 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 20:00:03.228911 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 20:00:03.248790 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 20:00:03.701581 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:00:04.273231 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20260809 20:00:04.304659 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 20:00:04.304662 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 20:00:04.329944 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 20:00:05.044289 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:00:05.045037 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20260809 20:00:05.381268 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 20:00:05.381378 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 20:00:05.403944 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 20:00:05.416664 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20260809 20:00:06.454154 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 20:00:06.454535 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 20:00:06.476557 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 20:00:06.698298 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20260809 20:00:07.526870 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260809 20:00:07.526934 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.3:46687, tserver@127.3.56.2:34843
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 20:00:07.546811 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 20:00:07.897085 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20260809 20:00:07.966145 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20260809 20:00:08.598609 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 20:00:08.599117 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 20:00:08.624078 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 20:00:09.005681 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:00:09.252641 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20260809 20:00:09.677306 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 20:00:09.677554 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 20:00:09.698441 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260809 20:00:10.050166 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20260809 20:00:10.554629 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20260809 20:00:10.657115 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20260809 20:00:10.748349 3692 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260809 20:00:10.749686 3869 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
20da41f8404d40e59d12b8d87f881a1d | 127.3.56.2:34843 | HEALTHY | <none> | 1 | 0
badc9e1d2ce547869d5d9ebf74a45b74 | 127.3.56.3:46687 | HEALTHY | <none> | 2 | 0
58b4bccc13254c8988cf60c89783f89a | 127.3.56.1:37591 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.3.56.1:37591: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.3.56.2:34843, 127.3.56.3:46687
ipki_server_key_size | 768 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
local_ip_for_outbound_sockets | 127.3.56.2 | experimental | 127.3.56.2:34843
local_ip_for_outbound_sockets | 127.3.56.3 | experimental | 127.3.56.3:46687
never_fsync | true | unsafe,advanced | 127.3.56.2:34843, 127.3.56.3:46687
openssl_security_level_override | 0 | unsafe,hidden | 127.3.56.2:34843, 127.3.56.3:46687
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_format | pb | hidden | 127.3.56.2:34843, 127.3.56.3:46687
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.56.2:34843
server_dump_info_path | /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.56.3:46687
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.3.56.2:34843, 127.3.56.3:46687
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+------------------------------------
builtin_ntp_servers | 127.3.56.20:44297 | 127.3.56.2:34843, 127.3.56.3:46687
time_source | builtin | 127.3.56.2:34843, 127.3.56.3:46687
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.3.56.62:33635, tserver@127.3.56.2:34843, tserver@127.3.56.3:46687
Tablet Summary
Tablet 97d3cd24f4b24b578165968c3aa82513 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
All reported replicas are:
A = 20da41f8404d40e59d12b8d87f881a1d
B = badc9e1d2ce547869d5d9ebf74a45b74
C = 58b4bccc13254c8988cf60c89783f89a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8375821116104cf999ae49469e06def0 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING [LEADER]
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 5814361c2f24496fa296e86169236f87 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): TS unavailable
20da41f8404d40e59d12b8d87f881a1d (127.3.56.2:34843): RUNNING [LEADER]
badc9e1d2ce547869d5d9ebf74a45b74 (127.3.56.3:46687): RUNNING
All reported replicas are:
A = 58b4bccc13254c8988cf60c89783f89a
B = 20da41f8404d40e59d12b8d87f881a1d
C = badc9e1d2ce547869d5d9ebf74a45b74
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 20:00:10.772820 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 20:00:11.757611 3331 maintenance_manager.cc:419] P 8b4ad58fdc334ba99ab671d1f7a6f422: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033375
W20260809 20:00:11.759639 3329 dir_manager.cc:275] Insufficient disk space under path /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskmEiWMh/test-tmp/all_types-itest.7.AllTypesItest_3.TestTimestampPadding.1786305491402637-3296-0/minicluster-data/master-0/data/data (488968192 bytes available vs 518351011 bytes reserved) (error 28)
W20260809 20:00:11.760268 3329 tablet_replica_mm_ops.cc:214] T 00000000000000000000000000000000 P 8b4ad58fdc334ba99ab671d1f7a6f422: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: tablet 00000000000000000000000000000000: unable to open output file for column entry_type INT8 NOT NULL: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 00000000000000000000000000000000's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
F20260809 20:00:11.760370 3329 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: tablet 00000000000000000000000000000000: unable to open output file for column entry_type INT8 NOT NULL: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 00000000000000000000000000000000's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
*** Check failure stack trace: ***
@ 0x7fec72ba3d5d google::LogMessage::Fail() at ??:0
@ 0x7fec72ba7b23 google::LogMessage::SendToLog() at ??:0
@ 0x7fec72ba375c google::LogMessage::Flush() at ??:0
@ 0x7fec72ba4ee9 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fec7662db22 kudu::tablet::FlushMRSOp::Perform() at ??:0
@ 0x7fec740c6cf2 kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7fec740c3e08 _ZZN4kudu18MaintenanceManager18RunSchedulerThreadEvENKUlvE_clEv at ??:0
@ 0x7fec740c8139 _ZNSt17_Function_handlerIFvvEZN4kudu18MaintenanceManager18RunSchedulerThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5645c12e6b9c std::function<>::operator()() at ??:0
@ 0x7fec74084767 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fec7408508f _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fec740867db _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5645c12e6b9c std::function<>::operator()() at ??:0
@ 0x7fec74074b52 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fec749f26db start_thread at ??:0
@ 0x7fec71d5a61f clone at ??:0
W20260809 20:00:11.847739 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20260809 20:00:12.191290 3775 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:00:12.270463 3960 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:00:12.809059 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260809 20:00:13.306203 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260809 20:00:14.187216 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:00:14.329391 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260809 20:00:15.337894 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:00:15.338771 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260809 20:00:15.753088 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260809 20:00:17.066718 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260809 20:00:17.898554 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260809 20:00:18.192700 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260809 20:00:19.593545 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:00:19.594231 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260809 20:00:20.666797 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260809 20:00:20.668411 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:00:20.669179 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260809 20:00:22.175071 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260809 20:00:23.210690 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260809 20:00:23.418654 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260809 20:00:24.544409 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260809 20:00:24.852802 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:00:25.855947 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260809 20:00:25.902763 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:00:25.903491 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260809 20:00:27.042137 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260809 20:00:28.041188 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260809 20:00:28.558542 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260809 20:00:29.675558 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260809 20:00:29.920746 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:00:30.366755 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260809 20:00:31.077306 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:00:31.078205 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260809 20:00:31.909803 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260809 20:00:32.798624 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260809 20:00:33.514005 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260809 20:00:34.266968 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260809 20:00:34.931509 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:00:35.408425 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260809 20:00:36.041115 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260809 20:00:36.481613 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:00:36.998350 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260809 20:00:37.930546 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260809 20:00:38.526947 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260809 20:00:39.671999 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260809 20:00:39.962872 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260809 20:00:40.495698 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260809 20:00:40.880975 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260809 20:00:41.845295 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:00:42.057103 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260809 20:00:42.957509 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260809 20:00:43.333676 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260809 20:00:44.559885 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260809 20:00:45.004503 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:00:45.135741 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260809 20:00:45.707427 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260809 20:00:47.033969 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260809 20:00:47.413079 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:00:47.444451 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260809 20:00:48.411310 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260809 20:00:49.465549 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260809 20:00:49.846783 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260809 20:00:50.259629 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:00:50.849823 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260809 20:00:51.960520 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260809 20:00:52.307814 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260809 20:00:52.978279 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:00:53.387005 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260809 20:00:54.421911 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260809 20:00:54.724083 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260809 20:00:55.406769 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:00:55.594398 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260809 20:00:56.667745 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260809 20:00:57.299322 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260809 20:00:58.035173 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:00:58.035944 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260809 20:00:59.269564 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260809 20:00:59.701162 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260809 20:01:00.545739 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260809 20:01:00.653000 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:01:01.607622 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260809 20:01:02.111140 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260809 20:01:03.185957 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:01:03.186815 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260809 20:01:03.945592 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260809 20:01:04.601244 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260809 20:01:05.759898 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260809 20:01:06.007369 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:01:06.619305 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260809 20:01:06.963488 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260809 20:01:08.220837 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:01:08.221735 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260809 20:01:08.921257 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260809 20:01:09.663118 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260809 20:01:10.786729 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260809 20:01:11.015841 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:01:11.578171 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260809 20:01:12.038977 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
8b4ad58fdc334ba99ab671d1f7a6f422 | 127.3.56.62:33635 | HEALTHY
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.56.20:44297 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
==================
Errors:
==================
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.56.62:33635, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
FAILED
I20260809 20:01:12.087225 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.56.62:33635, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:01:12.255749 3775 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (60 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:01:12.337404 3960 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (60 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:01:13.341894 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:01:13.342693 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260809 20:01:13.978018 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260809 20:01:14.581702 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260809 20:01:16.065421 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260809 20:01:16.379971 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:01:16.380686 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260809 20:01:16.959795 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260809 20:01:18.607722 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260809 20:01:18.667184 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:01:18.668022 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260809 20:01:19.478516 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260809 20:01:21.162477 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260809 20:01:21.220638 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260809 20:01:21.535184 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:01:22.069484 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260809 20:01:23.569290 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260809 20:01:23.607618 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260809 20:01:24.150457 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:01:24.733203 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260809 20:01:25.959777 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260809 20:01:26.145367 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260809 20:01:26.740298 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:01:27.194654 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260809 20:01:28.387722 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260809 20:01:28.608732 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260809 20:01:29.181710 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:01:29.634554 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260809 20:01:30.911947 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260809 20:01:31.256037 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260809 20:01:31.753568 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:01:32.377331 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260809 20:01:33.531566 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260809 20:01:33.941717 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260809 20:01:34.418670 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:01:34.928496 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260809 20:01:35.844957 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260809 20:01:36.360733 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260809 20:01:36.779943 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:01:37.518206 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260809 20:01:38.360423 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260809 20:01:39.051167 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260809 20:01:39.447072 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:01:40.229899 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260809 20:01:41.174053 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260809 20:01:41.408063 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260809 20:01:42.084929 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:01:42.604626 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260809 20:01:43.768193 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260809 20:01:43.768874 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260809 20:01:44.894819 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:01:45.080041 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260809 20:01:46.203078 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260809 20:01:46.305087 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260809 20:01:47.110713 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:01:47.733047 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260809 20:01:48.848732 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260809 20:01:49.131855 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260809 20:01:50.187783 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:01:50.308441 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260809 20:01:51.243216 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260809 20:01:51.566015 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260809 20:01:52.382691 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:01:52.811180 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260809 20:01:53.535050 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260809 20:01:54.086556 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260809 20:01:55.062579 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260809 20:01:55.639834 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:01:56.047894 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260809 20:01:56.553226 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260809 20:01:57.452682 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:01:57.453442 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260809 20:01:58.470266 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260809 20:01:59.019722 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260809 20:02:00.133630 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260809 20:02:01.124492 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:02:01.125212 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260809 20:02:01.754372 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260809 20:02:02.587368 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:02:02.788952 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260809 20:02:03.806154 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260809 20:02:04.063741 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260809 20:02:05.248307 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260809 20:02:06.279821 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:02:06.280831 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260809 20:02:06.594115 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260809 20:02:07.663056 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:02:07.664093 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260809 20:02:08.855897 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260809 20:02:09.196501 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260809 20:02:10.063302 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260809 20:02:11.625465 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260809 20:02:11.752115 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:02:11.752916 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260809 20:02:12.323767 3775 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (120 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:02:12.406011 3960 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (120 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:02:12.568547 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260809 20:02:12.988654 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
Master Summary
UUID | Address | Status
-------------------------------+-------------------+-------------
<unknown> (127.3.56.62:33635) | 127.3.56.62:33635 | UNAVAILABLE
Error from 127.3.56.62:33635: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.56.62:33635, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
FAILED
I20260809 20:02:13.187512 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.56.62:33635, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:02:13.936796 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260809 20:02:14.227752 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260809 20:02:15.243019 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260809 20:02:16.047466 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260809 20:02:16.818688 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:02:16.819588 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260809 20:02:17.812635 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260809 20:02:18.332520 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:02:18.333282 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260809 20:02:19.184990 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260809 20:02:20.421957 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260809 20:02:20.502262 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260809 20:02:21.683974 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260809 20:02:22.281945 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:02:22.825310 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260809 20:02:23.030071 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260809 20:02:23.507957 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:02:24.278550 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260809 20:02:25.171401 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260809 20:02:25.570123 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260809 20:02:26.627789 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260809 20:02:27.624156 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260809 20:02:27.823117 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:02:28.045500 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260809 20:02:28.783020 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:02:29.284981 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260809 20:02:30.158679 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260809 20:02:30.429750 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260809 20:02:32.133812 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260809 20:02:32.546929 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260809 20:02:32.720188 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260809 20:02:33.202859 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:02:34.229547 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:02:34.547310 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260809 20:02:35.089243 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260809 20:02:35.185717 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260809 20:02:36.890607 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260809 20:02:37.670231 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260809 20:02:37.957988 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260809 20:02:38.476379 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:02:39.282373 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:02:39.400841 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260809 20:02:40.314827 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260809 20:02:40.329479 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260809 20:02:42.026212 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260809 20:02:42.938153 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260809 20:02:42.960089 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260809 20:02:43.530088 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:02:44.395104 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:02:44.573621 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260809 20:02:45.348335 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260809 20:02:45.398200 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260809 20:02:47.146664 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260809 20:02:47.813524 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260809 20:02:47.921000 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260809 20:02:48.616490 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:02:49.400456 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260809 20:02:49.540460 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260809 20:02:50.393579 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260809 20:02:50.531471 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260809 20:02:52.154776 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260809 20:02:52.905154 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260809 20:02:52.937287 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260809 20:02:53.780064 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:02:54.609652 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:02:54.680299 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260809 20:02:55.519064 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260809 20:02:55.766295 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260809 20:02:57.196328 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260809 20:02:57.908490 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260809 20:02:58.338330 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260809 20:02:59.231530 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:02:59.679330 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260809 20:02:59.942436 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:03:00.471539 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260809 20:03:00.477763 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260809 20:03:02.384229 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260809 20:03:03.021494 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260809 20:03:03.315004 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260809 20:03:04.234457 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:03:04.681428 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260809 20:03:05.095173 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:03:05.565652 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260809 20:03:05.858393 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260809 20:03:07.234171 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260809 20:03:07.908212 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260809 20:03:08.309816 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260809 20:03:09.239622 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:03:09.640250 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260809 20:03:10.495754 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:03:10.588140 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260809 20:03:10.880429 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260809 20:03:12.350536 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260809 20:03:12.394292 3775 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (180 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:03:12.475485 3960 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (180 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:03:12.995887 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20260809 20:03:13.247264 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
-------------------------------+-------------------+-------------
<unknown> (127.3.56.62:33635) | 127.3.56.62:33635 | UNAVAILABLE
Error from 127.3.56.62:33635: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.56.62:33635, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
FAILED
I20260809 20:03:14.311841 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.56.62:33635, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:03:14.675874 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:03:14.676776 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20260809 20:03:15.156620 8944 consensus_queue.cc:579] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d [LEADER]: Leader has been unable to successfully communicate with peer 58b4bccc13254c8988cf60c89783f89a for more than 300 seconds (300.119s)
I20260809 20:03:15.252568 8765 consensus_queue.cc:579] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 [LEADER]: Leader has been unable to successfully communicate with peer 58b4bccc13254c8988cf60c89783f89a for more than 300 seconds (300.262s)
I20260809 20:03:15.443372 8949 consensus_queue.cc:579] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 [LEADER]: Leader has been unable to successfully communicate with peer 58b4bccc13254c8988cf60c89783f89a for more than 300 seconds (300.335s)
W20260809 20:03:15.444765 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260809 20:03:15.724272 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:03:15.725023 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260809 20:03:17.227442 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260809 20:03:17.674679 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260809 20:03:18.232770 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260809 20:03:19.869876 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:03:19.870647 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260809 20:03:20.478776 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260809 20:03:20.684373 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260809 20:03:20.887418 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:03:22.257499 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260809 20:03:22.915066 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260809 20:03:23.328400 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260809 20:03:24.862751 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260809 20:03:25.252349 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260809 20:03:25.456650 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:03:25.800467 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260809 20:03:26.283134 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:03:27.512521 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260809 20:03:27.895519 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260809 20:03:28.399681 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260809 20:03:30.332556 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260809 20:03:30.602396 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260809 20:03:30.819489 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:03:30.915983 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260809 20:03:31.496886 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:03:33.066257 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260809 20:03:33.277019 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260809 20:03:33.612593 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260809 20:03:35.723641 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260809 20:03:35.768440 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260809 20:03:35.952574 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260809 20:03:36.346242 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:03:36.925760 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:03:38.343379 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260809 20:03:38.384632 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260809 20:03:38.483879 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260809 20:03:40.767499 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260809 20:03:40.837670 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260809 20:03:41.131067 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260809 20:03:41.356230 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:03:41.988495 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:03:43.257320 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260809 20:03:43.488686 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260809 20:03:43.630234 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260809 20:03:45.864384 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260809 20:03:46.053228 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260809 20:03:46.135648 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260809 20:03:46.371678 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:03:47.073074 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:03:48.256014 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260809 20:03:48.321883 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260809 20:03:48.582000 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260809 20:03:50.669773 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260809 20:03:50.931883 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260809 20:03:51.366925 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260809 20:03:51.638223 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:03:52.276515 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:03:53.207444 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260809 20:03:53.374811 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260809 20:03:53.794953 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260809 20:03:55.901012 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260809 20:03:55.999769 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260809 20:03:56.242275 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260809 20:03:56.847568 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:03:57.554243 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:03:58.131827 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260809 20:03:58.521390 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260809 20:03:58.542335 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260809 20:04:00.542620 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260809 20:04:01.006779 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260809 20:04:01.050467 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260809 20:04:02.137136 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:04:02.897935 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:04:03.157514 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260809 20:04:03.371917 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260809 20:04:03.745380 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260809 20:04:05.464815 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260809 20:04:05.766660 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260809 20:04:06.084223 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260809 20:04:07.563619 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:04:08.016360 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260809 20:04:08.097601 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:04:08.133154 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260809 20:04:08.547036 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260809 20:04:10.366207 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260809 20:04:10.753612 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260809 20:04:11.085597 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260809 20:04:12.509367 3960 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (241 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:04:12.927946 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:04:12.928818 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260809 20:04:13.220307 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:04:13.221083 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260809 20:04:13.224460 3775 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (241 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:04:13.613513 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
-------------------------------+-------------------+-------------
<unknown> (127.3.56.62:33635) | 127.3.56.62:33635 | UNAVAILABLE
Error from 127.3.56.62:33635: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.56.62:33635, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
FAILED
I20260809 20:04:15.405625 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.56.62:33635, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:04:15.465205 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260809 20:04:15.488075 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260809 20:04:16.081084 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260809 20:04:17.946339 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:04:17.947178 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260809 20:04:18.150174 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260809 20:04:18.522833 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:04:18.523521 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260809 20:04:20.633098 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260809 20:04:20.761328 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260809 20:04:21.127352 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260809 20:04:23.109844 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:04:23.110652 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260809 20:04:23.197233 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260809 20:04:23.548233 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:04:23.548982 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260809 20:04:25.659399 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260809 20:04:25.774925 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260809 20:04:26.106786 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260809 20:04:28.133996 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260809 20:04:28.477593 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:04:28.478416 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260809 20:04:28.512960 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260809 20:04:28.630718 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:04:30.600471 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260809 20:04:30.784282 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260809 20:04:31.382920 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260809 20:04:32.960197 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260809 20:04:33.344794 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260809 20:04:33.514530 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20260809 20:04:33.911033 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:04:34.021831 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260809 20:04:35.476491 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260809 20:04:35.887137 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260809 20:04:36.473292 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260809 20:04:37.897550 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260809 20:04:38.491132 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260809 20:04:38.918046 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:04:38.919056 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:04:38.919736 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260809 20:04:40.332612 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260809 20:04:40.938796 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260809 20:04:41.706226 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260809 20:04:42.918900 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260809 20:04:43.415762 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260809 20:04:44.162505 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:04:44.163388 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260809 20:04:44.257863 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:04:45.180593 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260809 20:04:46.042518 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260809 20:04:46.931365 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260809 20:04:47.746232 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260809 20:04:48.727919 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260809 20:04:49.294044 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:04:49.295116 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260809 20:04:49.502115 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:04:49.914987 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260809 20:04:50.942114 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260809 20:04:51.812675 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260809 20:04:52.090415 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260809 20:04:53.253832 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260809 20:04:54.259513 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260809 20:04:54.508896 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:04:54.509774 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260809 20:04:54.827564 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:04:55.871146 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260809 20:04:56.597204 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260809 20:04:56.854159 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260809 20:04:58.639823 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260809 20:04:58.961973 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260809 20:04:59.384063 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260809 20:04:59.530499 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:04:59.844290 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:05:00.947624 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260809 20:05:01.316741 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260809 20:05:01.865633 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260809 20:05:03.431342 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260809 20:05:03.938549 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260809 20:05:04.286059 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260809 20:05:04.866236 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:05:05.373647 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:05:05.894454 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260809 20:05:06.352383 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260809 20:05:06.988260 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260809 20:05:08.256080 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260809 20:05:08.621150 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260809 20:05:09.485811 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260809 20:05:09.975234 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:05:10.431772 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:05:10.716012 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260809 20:05:10.985746 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260809 20:05:11.719542 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260809 20:05:12.576666 3960 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (301 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:05:12.939929 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260809 20:05:13.293941 3775 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (301 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:05:13.666435 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260809 20:05:14.521519 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260809 20:05:15.035122 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:05:15.352547 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260809 20:05:15.636521 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:05:15.987421 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
-------------------------------+-------------------+-------------
<unknown> (127.3.56.62:33635) | 127.3.56.62:33635 | UNAVAILABLE
Error from 127.3.56.62:33635: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.56.62:33635, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
FAILED
I20260809 20:05:16.506330 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.56.62:33635, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:05:17.005007 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260809 20:05:17.760768 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260809 20:05:18.431017 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260809 20:05:19.367161 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260809 20:05:19.970434 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260809 20:05:20.281373 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:05:20.775810 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:05:21.454959 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260809 20:05:21.775108 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260809 20:05:22.113619 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260809 20:05:23.879910 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260809 20:05:24.188973 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260809 20:05:24.782178 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260809 20:05:25.603884 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:05:26.121905 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:05:26.578404 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260809 20:05:26.671851 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260809 20:05:27.364248 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260809 20:05:29.230293 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260809 20:05:29.347455 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260809 20:05:29.918733 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260809 20:05:30.936910 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:05:31.453080 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:05:31.453840 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260809 20:05:31.779273 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260809 20:05:32.600759 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260809 20:05:34.073103 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260809 20:05:34.300662 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260809 20:05:34.928172 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260809 20:05:35.948176 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:05:36.494573 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:05:36.495499 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260809 20:05:36.706746 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260809 20:05:37.382522 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260809 20:05:39.022953 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260809 20:05:39.479028 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260809 20:05:39.901243 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260809 20:05:41.071635 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:05:41.387285 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260809 20:05:41.893553 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:05:41.959095 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260809 20:05:42.584501 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260809 20:05:43.656698 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260809 20:05:44.716454 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260809 20:05:45.050402 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260809 20:05:46.190482 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:05:46.265331 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260809 20:05:47.172545 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:05:47.358875 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260809 20:05:47.725912 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260809 20:05:48.615461 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260809 20:05:49.682610 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260809 20:05:50.206964 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260809 20:05:50.750523 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260809 20:05:51.333096 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:05:52.234056 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260809 20:05:52.513302 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20260809 20:05:52.758980 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260809 20:05:53.096598 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260809 20:05:54.590689 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260809 20:05:55.295529 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260809 20:05:55.673586 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260809 20:05:56.459617 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:05:56.982539 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260809 20:05:57.741107 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:05:57.934405 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260809 20:05:58.357564 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260809 20:05:59.524793 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260809 20:06:00.387426 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260809 20:06:00.792443 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260809 20:06:01.490504 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:06:02.058692 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260809 20:06:02.965330 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:06:03.144120 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260809 20:06:03.393566 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260809 20:06:04.467372 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260809 20:06:05.642510 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260809 20:06:05.923996 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260809 20:06:06.550688 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:06:07.109261 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260809 20:06:08.156836 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260809 20:06:08.170361 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:06:08.171041 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260809 20:06:09.385568 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260809 20:06:10.451570 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260809 20:06:10.654502 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260809 20:06:11.613867 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:06:11.614565 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260809 20:06:12.644160 3960 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (361 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:06:13.033062 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260809 20:06:13.116569 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260809 20:06:13.364553 3775 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (361 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:06:13.428107 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:06:13.915755 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260809 20:06:15.441773 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260809 20:06:15.520031 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260809 20:06:16.713539 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:06:16.714308 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
-------------------------------+-------------------+-------------
<unknown> (127.3.56.62:33635) | 127.3.56.62:33635 | UNAVAILABLE
Error from 127.3.56.62:33635: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.56.62:33635, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
FAILED
I20260809 20:06:17.580587 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.56.62:33635, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:06:17.672236 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260809 20:06:17.783898 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260809 20:06:18.725368 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:06:19.373878 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260809 20:06:20.085561 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260809 20:06:20.304203 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260809 20:06:21.895011 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:06:21.895789 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260809 20:06:22.752507 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260809 20:06:23.185089 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260809 20:06:24.074354 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:06:24.607306 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260809 20:06:25.235472 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260809 20:06:25.682217 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260809 20:06:26.900864 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20260809 20:06:27.595877 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260809 20:06:27.793883 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260809 20:06:28.609031 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260809 20:06:29.405230 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:06:30.218220 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260809 20:06:30.360036 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260809 20:06:31.113600 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260809 20:06:32.130131 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:06:32.710172 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260809 20:06:32.812237 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260809 20:06:33.844209 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260809 20:06:34.615234 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:06:35.157428 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260809 20:06:35.279467 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260809 20:06:36.169875 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260809 20:06:37.499214 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:06:37.840238 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260809 20:06:38.042309 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260809 20:06:38.596843 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260809 20:06:39.675403 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:06:40.054915 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260809 20:06:40.472747 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260809 20:06:41.030961 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260809 20:06:42.406190 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260809 20:06:42.812193 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:06:42.812969 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260809 20:06:43.443509 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260809 20:06:45.039719 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:06:45.040639 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260809 20:06:45.573727 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260809 20:06:45.935240 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260809 20:06:47.750063 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260809 20:06:47.862700 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:06:47.948894 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260809 20:06:48.349380 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260809 20:06:50.438177 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260809 20:06:50.566416 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:06:50.567270 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260809 20:06:51.019726 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260809 20:06:52.819857 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260809 20:06:52.986159 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260809 20:06:53.043701 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:06:53.643893 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260809 20:06:55.362393 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260809 20:06:55.566442 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:06:55.567317 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260809 20:06:56.176251 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260809 20:06:57.807086 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260809 20:06:57.993762 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260809 20:06:58.350649 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:06:58.905300 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260809 20:07:00.332335 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260809 20:07:00.558679 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260809 20:07:01.006379 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:07:01.400493 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260809 20:07:02.711997 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260809 20:07:03.010767 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260809 20:07:03.453222 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:07:03.815179 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260809 20:07:05.047611 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260809 20:07:05.207808 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260809 20:07:06.049868 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:07:06.169759 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260809 20:07:07.244063 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260809 20:07:07.772535 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260809 20:07:08.766834 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:07:08.767588 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260809 20:07:09.812402 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260809 20:07:10.379743 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260809 20:07:11.090602 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260809 20:07:11.491741 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:07:12.399236 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260809 20:07:12.711921 3960 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (421 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:07:13.069272 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260809 20:07:13.209883 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260809 20:07:13.432122 3775 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (421 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:07:13.780874 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:07:14.651901 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260809 20:07:15.777146 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260809 20:07:16.033279 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260809 20:07:16.593485 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:07:16.981231 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260809 20:07:18.527644 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260809 20:07:18.584673 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
-------------------------------+-------------------+-------------
<unknown> (127.3.56.62:33635) | 127.3.56.62:33635 | UNAVAILABLE
Error from 127.3.56.62:33635: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.56.62:33635, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
FAILED
I20260809 20:07:18.766657 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.56.62:33635, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:07:18.985164 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260809 20:07:19.722754 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260809 20:07:21.117177 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260809 20:07:21.125166 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260809 20:07:21.806439 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:07:22.301512 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260809 20:07:23.537617 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260809 20:07:23.747896 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260809 20:07:24.094955 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:07:24.770943 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260809 20:07:26.113050 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260809 20:07:26.291884 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260809 20:07:27.364766 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:07:27.365579 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260809 20:07:28.796228 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260809 20:07:28.848723 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260809 20:07:29.225253 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:07:29.949956 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260809 20:07:31.359793 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260809 20:07:31.483481 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260809 20:07:32.422914 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:07:32.423647 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260809 20:07:33.691011 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260809 20:07:34.042944 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260809 20:07:34.581228 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:07:34.956676 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260809 20:07:36.022464 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260809 20:07:36.662981 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260809 20:07:37.405447 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260809 20:07:37.981763 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:07:38.581995 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260809 20:07:38.890318 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260809 20:07:39.855376 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:07:40.009956 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260809 20:07:41.117484 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260809 20:07:41.265240 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260809 20:07:42.462306 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260809 20:07:43.457626 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:07:43.556504 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260809 20:07:44.004702 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260809 20:07:44.731379 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260809 20:07:45.058990 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:07:45.680998 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260809 20:07:46.618377 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260809 20:07:47.499708 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260809 20:07:48.430225 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260809 20:07:48.545790 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:07:49.096422 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260809 20:07:50.176271 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:07:50.185082 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260809 20:07:51.077880 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260809 20:07:51.914951 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260809 20:07:53.146157 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260809 20:07:53.507936 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260809 20:07:53.709316 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20260809 20:07:54.447398 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260809 20:07:55.367741 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:07:55.617843 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260809 20:07:55.967774 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260809 20:07:56.930011 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260809 20:07:58.257529 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260809 20:07:58.582787 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260809 20:07:58.737214 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:07:59.342330 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260809 20:08:00.459345 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:08:00.583653 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260809 20:08:01.302683 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260809 20:08:01.904093 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260809 20:08:03.091002 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260809 20:08:03.822983 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260809 20:08:03.989537 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:08:04.309875 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260809 20:08:05.471346 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260809 20:08:05.667809 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:08:06.255702 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260809 20:08:06.839298 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260809 20:08:07.707574 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260809 20:08:08.577687 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260809 20:08:09.226495 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260809 20:08:09.497722 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:08:10.575667 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260809 20:08:10.728590 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:08:11.120945 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260809 20:08:11.829026 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260809 20:08:12.778227 3960 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (481 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:08:13.170729 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260809 20:08:13.502030 3775 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (481 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:08:13.694625 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260809 20:08:14.375264 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260809 20:08:14.577864 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:08:15.631942 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260809 20:08:15.923693 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:08:15.924453 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260809 20:08:17.108037 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260809 20:08:18.114131 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260809 20:08:18.435132 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260809 20:08:19.600327 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
-------------------------------+-------------------+-------------
<unknown> (127.3.56.62:33635) | 127.3.56.62:33635 | UNAVAILABLE
Error from 127.3.56.62:33635: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.56.62:33635, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
FAILED
I20260809 20:08:19.885906 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.56.62:33635, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:08:20.027177 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:08:20.415340 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260809 20:08:20.891382 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260809 20:08:21.241305 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:08:22.273556 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260809 20:08:22.684258 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260809 20:08:23.417971 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260809 20:08:24.572396 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260809 20:08:25.107353 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:08:25.108147 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260809 20:08:25.844439 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260809 20:08:26.606218 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:08:27.081810 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260809 20:08:27.768114 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260809 20:08:28.314955 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260809 20:08:29.733898 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260809 20:08:30.009527 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260809 20:08:30.502230 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:08:30.719292 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260809 20:08:31.755291 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:08:32.320565 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260809 20:08:32.465188 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260809 20:08:33.386272 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260809 20:08:34.908687 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260809 20:08:35.146638 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260809 20:08:35.552949 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:08:35.905614 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260809 20:08:36.911968 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:08:37.473816 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260809 20:08:37.634426 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260809 20:08:38.444826 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260809 20:08:39.988281 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260809 20:08:40.292089 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260809 20:08:40.771629 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:08:41.011487 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260809 20:08:42.351806 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:08:42.352681 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260809 20:08:42.751608 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260809 20:08:43.275890 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260809 20:08:44.632402 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260809 20:08:45.426692 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260809 20:08:45.784240 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260809 20:08:45.906767 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:08:47.065672 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260809 20:08:47.394266 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:08:47.938531 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260809 20:08:48.358783 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260809 20:08:49.464406 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260809 20:08:50.601229 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260809 20:08:50.815225 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260809 20:08:51.089654 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:08:51.774781 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260809 20:08:52.864940 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:08:53.002693 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260809 20:08:53.331665 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260809 20:08:54.289913 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260809 20:08:55.512313 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260809 20:08:55.803289 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260809 20:08:56.575258 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:08:56.714656 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260809 20:08:57.963155 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:08:58.079084 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260809 20:08:58.512108 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260809 20:08:59.297940 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260809 20:09:00.449507 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260809 20:09:00.973308 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260809 20:09:01.689170 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260809 20:09:02.080994 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:09:02.964488 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:09:03.087405 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260809 20:09:03.517997 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260809 20:09:04.232928 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260809 20:09:05.442049 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260809 20:09:06.058223 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260809 20:09:06.768661 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260809 20:09:07.107419 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:09:08.055105 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260809 20:09:08.442377 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:09:08.457671 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260809 20:09:09.511018 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260809 20:09:10.618666 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260809 20:09:10.933804 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260809 20:09:11.903046 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260809 20:09:12.285840 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:09:12.843918 3960 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (541 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:09:13.325215 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260809 20:09:13.357470 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260809 20:09:13.572750 3775 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (541 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:09:13.833674 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:09:14.278680 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260809 20:09:16.046594 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260809 20:09:16.128178 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260809 20:09:16.724776 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260809 20:09:17.655893 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:09:18.321764 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260809 20:09:18.694319 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260809 20:09:19.313452 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:09:19.314314 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260809 20:09:20.716742 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
-------------------------------+-------------------+-------------
<unknown> (127.3.56.62:33635) | 127.3.56.62:33635 | UNAVAILABLE
Error from 127.3.56.62:33635: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.56.62:33635, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
FAILED
I20260809 20:09:20.957059 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.56.62:33635, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:09:21.300454 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260809 20:09:21.981997 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260809 20:09:22.896608 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:09:23.291290 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260809 20:09:23.839040 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260809 20:09:24.539535 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:09:24.540319 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260809 20:09:25.597447 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260809 20:09:26.063828 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260809 20:09:26.853302 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260809 20:09:28.070441 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:09:28.110437 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260809 20:09:28.670962 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260809 20:09:29.293061 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260809 20:09:29.651273 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:09:30.481722 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260809 20:09:30.866217 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260809 20:09:31.830859 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260809 20:09:33.080701 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260809 20:09:33.373152 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:09:33.374048 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260809 20:09:34.136269 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260809 20:09:34.738220 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:09:35.494346 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260809 20:09:35.920869 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260809 20:09:36.737727 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260809 20:09:38.132673 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260809 20:09:38.310416 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260809 20:09:38.700623 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:09:39.111184 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260809 20:09:39.874490 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:09:40.315989 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260809 20:09:40.453847 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260809 20:09:41.668893 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260809 20:09:42.742879 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260809 20:09:42.869129 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260809 20:09:43.701531 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:09:44.008078 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260809 20:09:45.150415 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:09:45.254812 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260809 20:09:45.288527 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260809 20:09:46.657845 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260809 20:09:47.712682 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260809 20:09:47.885115 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260809 20:09:48.843298 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:09:49.164140 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260809 20:09:50.148499 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260809 20:09:50.193111 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:09:50.193830 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260809 20:09:51.775863 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260809 20:09:52.685740 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260809 20:09:52.795718 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260809 20:09:54.141944 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260809 20:09:54.263442 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:09:55.172914 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260809 20:09:55.279196 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260809 20:09:55.478355 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:09:56.408635 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260809 20:09:57.370895 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260809 20:09:57.826305 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260809 20:09:58.881026 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260809 20:09:59.372855 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:09:59.873809 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260809 20:10:00.388486 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260809 20:10:00.875252 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:10:01.498114 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260809 20:10:02.416137 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260809 20:10:02.786463 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260809 20:10:03.875650 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260809 20:10:04.725981 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:10:04.872853 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260809 20:10:05.232306 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260809 20:10:06.023684 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:10:06.463471 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260809 20:10:07.432694 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260809 20:10:07.616230 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260809 20:10:09.128221 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260809 20:10:10.054598 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260809 20:10:10.298900 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:10:10.299850 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260809 20:10:11.250595 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260809 20:10:11.795612 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260809 20:10:12.640198 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260809 20:10:12.914881 3960 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (601 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:10:13.029142 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260809 20:10:13.642191 3775 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (601 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:10:14.355417 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260809 20:10:15.105633 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260809 20:10:15.408908 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260809 20:10:15.486954 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:10:16.449000 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:10:16.721081 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260809 20:10:17.527220 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260809 20:10:17.869668 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260809 20:10:19.404400 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260809 20:10:20.078730 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260809 20:10:20.597433 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260809 20:10:20.622345 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:10:21.604331 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
Master Summary
UUID | Address | Status
-------------------------------+-------------------+-------------
<unknown> (127.3.56.62:33635) | 127.3.56.62:33635 | UNAVAILABLE
Error from 127.3.56.62:33635: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.56.62:33635, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
FAILED
I20260809 20:10:22.021212 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.56.62:33635, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:10:22.168455 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260809 20:10:22.631148 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260809 20:10:22.752264 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260809 20:10:24.552933 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260809 20:10:25.244315 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260809 20:10:25.361159 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260809 20:10:25.742900 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:10:26.640152 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:10:27.321671 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260809 20:10:27.680827 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260809 20:10:27.843879 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260809 20:10:29.861198 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260809 20:10:30.078864 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260809 20:10:30.254528 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260809 20:10:30.749925 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:10:31.753902 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:10:32.342152 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260809 20:10:32.466885 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260809 20:10:32.795706 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260809 20:10:34.485496 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260809 20:10:34.740953 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260809 20:10:35.378849 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260809 20:10:35.821751 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:10:37.115751 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:10:37.116528 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260809 20:10:37.169270 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260809 20:10:37.958943 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260809 20:10:39.502204 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260809 20:10:39.688834 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260809 20:10:40.561172 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260809 20:10:41.006656 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:10:41.979017 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260809 20:10:42.123526 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:10:42.124303 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260809 20:10:42.842518 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260809 20:10:44.374382 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260809 20:10:44.739797 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260809 20:10:45.187372 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260809 20:10:46.118266 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:10:47.108906 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260809 20:10:47.119975 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260809 20:10:47.570703 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260809 20:10:47.712244 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:10:49.391893 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260809 20:10:49.742770 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260809 20:10:49.850135 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260809 20:10:51.239034 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:10:51.968711 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260809 20:10:52.219036 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260809 20:10:52.244349 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260809 20:10:52.888921 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:10:54.336499 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260809 20:10:54.626734 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260809 20:10:54.640149 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260809 20:10:56.317601 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:10:56.774732 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260809 20:10:57.321818 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260809 20:10:57.344588 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260809 20:10:58.151310 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:10:59.121687 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260809 20:10:59.497642 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260809 20:10:59.690169 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260809 20:11:01.358892 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:11:01.367206 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260809 20:11:01.901278 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260809 20:11:02.072978 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260809 20:11:03.263419 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:11:03.861063 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260809 20:11:04.695683 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260809 20:11:04.842332 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260809 20:11:06.359244 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260809 20:11:06.597172 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:11:07.201166 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260809 20:11:07.407353 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260809 20:11:08.294492 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:11:09.134546 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260809 20:11:09.721755 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260809 20:11:09.883592 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260809 20:11:11.777221 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260809 20:11:11.812052 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:11:12.193944 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260809 20:11:12.202606 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260809 20:11:12.983162 3960 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (661 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:11:13.306885 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260809 20:11:13.708389 3775 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (661 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:11:14.566901 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260809 20:11:14.708971 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260809 20:11:14.817530 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260809 20:11:16.891814 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260809 20:11:17.094830 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:11:17.214804 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260809 20:11:17.619145 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260809 20:11:18.740188 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:11:19.357426 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260809 20:11:19.840529 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260809 20:11:20.062932 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260809 20:11:21.570585 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260809 20:11:22.290117 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260809 20:11:22.321264 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:11:22.322206 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
-------------------------------+-------------------+-------------
<unknown> (127.3.56.62:33635) | 127.3.56.62:33635 | UNAVAILABLE
Error from 127.3.56.62:33635: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.56.62:33635, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
FAILED
I20260809 20:11:23.087399 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.56.62:33635, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:11:23.812108 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:11:24.162400 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260809 20:11:24.879238 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260809 20:11:24.912439 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260809 20:11:26.761359 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260809 20:11:27.420168 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260809 20:11:27.525599 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:11:27.526305 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260809 20:11:28.967530 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260809 20:11:29.562543 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260809 20:11:29.877692 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260809 20:11:30.079176 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260809 20:11:32.111989 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260809 20:11:32.345055 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260809 20:11:32.565558 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:11:32.566371 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260809 20:11:34.157420 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:11:34.583880 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260809 20:11:34.633080 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260809 20:11:34.751164 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260809 20:11:37.111140 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260809 20:11:37.262152 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260809 20:11:37.442162 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260809 20:11:37.844973 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:11:39.289678 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:11:39.392081 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260809 20:11:39.896245 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260809 20:11:39.964480 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260809 20:11:41.675529 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260809 20:11:42.271700 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260809 20:11:42.849824 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260809 20:11:43.375106 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:11:44.012934 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260809 20:11:44.407197 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:11:44.949700 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260809 20:11:45.112151 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260809 20:11:46.535539 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260809 20:11:47.362674 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260809 20:11:47.749889 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260809 20:11:48.723202 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:11:48.909866 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260809 20:11:49.685670 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260809 20:11:49.883705 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:11:50.362537 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260809 20:11:51.495151 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260809 20:11:52.207276 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260809 20:11:52.599960 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260809 20:11:53.791055 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260809 20:11:54.260066 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:11:54.699817 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260809 20:11:55.015369 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:11:55.016129 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260809 20:11:56.421921 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260809 20:11:57.428305 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260809 20:11:57.551277 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260809 20:11:58.932401 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260809 20:11:59.272615 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:11:59.722867 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260809 20:12:00.130941 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:12:00.131677 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260809 20:12:01.386942 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260809 20:12:02.156003 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260809 20:12:02.670743 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260809 20:12:03.569103 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260809 20:12:04.671073 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:12:04.671984 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260809 20:12:05.024784 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260809 20:12:05.514776 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:12:05.987228 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260809 20:12:07.131927 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260809 20:12:07.405151 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260809 20:12:08.435187 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260809 20:12:09.512283 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260809 20:12:09.805895 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260809 20:12:10.028223 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:12:10.517882 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:12:10.972723 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260809 20:12:11.989112 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260809 20:12:12.527657 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260809 20:12:13.049901 3960 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (721 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:12:13.386250 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260809 20:12:13.776427 3775 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (721 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:12:14.609212 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260809 20:12:14.917569 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260809 20:12:15.494441 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:12:15.625669 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:12:16.243866 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260809 20:12:17.040383 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260809 20:12:17.369212 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260809 20:12:18.588624 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260809 20:12:19.084312 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260809 20:12:19.880828 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260809 20:12:20.802736 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:12:20.804394 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:12:20.991858 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260809 20:12:21.964586 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260809 20:12:22.559805 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260809 20:12:23.406149 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
-------------------------------+-------------------+-------------
<unknown> (127.3.56.62:33635) | 127.3.56.62:33635 | UNAVAILABLE
Error from 127.3.56.62:33635: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.56.62:33635, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
FAILED
I20260809 20:12:24.155037 3296 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.56.62:33635, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
W20260809 20:12:24.300976 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260809 20:12:24.927554 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260809 20:12:26.062279 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:12:26.232182 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:12:26.232897 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260809 20:12:27.134727 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260809 20:12:27.364889 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260809 20:12:28.627988 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260809 20:12:29.932036 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260809 20:12:30.142189 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260809 20:12:31.372814 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:12:31.405989 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:12:31.429693 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260809 20:12:32.311267 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260809 20:12:32.404035 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260809 20:12:34.214280 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260809 20:12:34.604297 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260809 20:12:34.805186 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260809 20:12:36.557276 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:12:36.558135 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260809 20:12:36.835142 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:12:37.342900 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260809 20:12:37.356110 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260809 20:12:39.146430 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260809 20:12:39.767288 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260809 20:12:40.175917 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260809 20:12:41.665031 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
************************ BEGIN STACKS **************************
W20260809 20:12:41.757068 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260809 20:12:41.909785 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
[New LWP 3301]
[New LWP 3302]
[New LWP 3303]
[New LWP 3963]
[New LWP 3965]
[New LWP 3966]
[New LWP 3969]
[New LWP 10453]
[New LWP 10456]
[New LWP 10457]
[New LWP 10458]
[New LWP 10463]
[New LWP 10464]
[New LWP 10465]
[New LWP 10467]
[New LWP 10469]
W20260809 20:12:42.228664 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260809 20:12:42.392845 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260809 20:12:44.208666 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
W20260809 20:12:45.058525 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260809 20:12:45.237174 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260809 20:12:46.709587 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:12:46.710340 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260809 20:12:47.237366 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:12:47.381451 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260809 20:12:47.801828 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260809 20:12:49.219898 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260809 20:12:49.927340 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260809 20:12:50.388501 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260809 20:12:51.724278 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260809 20:12:51.801826 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260809 20:12:52.102383 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260809 20:12:52.442454 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:12:53.050231 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260809 20:12:53.974159 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260809 20:12:54.685588 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260809 20:12:55.682130 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260809 20:12:56.472819 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260809 20:12:57.009272 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260809 20:12:57.062143 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260809 20:12:57.704797 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:12:58.109123 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260809 20:12:58.968451 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260809 20:12:59.737105 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260809 20:13:00.619024 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260809 20:13:01.273878 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260809 20:13:02.218549 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260809 20:13:02.219321 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
W20260809 20:13:03.268165 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260809 20:13:03.269109 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260809 20:13:03.850489 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
W20260809 20:13:04.737758 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
W20260809 20:13:05.893402 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
W20260809 20:13:06.275391 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
W20260809 20:13:07.127458 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
W20260809 20:13:07.569283 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 21 similar messages]
0x00007f50ec82bad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
Id Target Id Frame
* 1 Thread 0x7f50dfb0d280 (LWP 3296) "all_types-itest" 0x00007f50ec82bad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
2 Thread 0x7f50ddae8700 (LWP 3301) "all_types-itest" 0x00007f50ec83032a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
3 Thread 0x7f50dc10e700 (LWP 3302) "kernel-watcher-" 0x00007f50ec82bfb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
4 Thread 0x7f50db90d700 (LWP 3303) "rpc reactor-330" 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
5 Thread 0x7f50db10c700 (LWP 3963) "rpc reactor-396" 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
6 Thread 0x7f50da704700 (LWP 3965) "rpc reactor-396" 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
7 Thread 0x7f50d9f03700 (LWP 3966) "rpc reactor-396" 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
8 Thread 0x7f50d9702700 (LWP 3969) "rpc reactor-396" 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
9 Thread 0x7f50d76fe700 (LWP 10453) "rpc reactor-104" 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
10 Thread 0x7f50d2ef5700 (LWP 10456) "rpc reactor-104" 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
11 Thread 0x7f50d26f4700 (LWP 10457) "rpc reactor-104" 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
12 Thread 0x7f50d36f6700 (LWP 10458) "rpc reactor-104" 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
13 Thread 0x7f50d7eff700 (LWP 10463) "rpc reactor-104" 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
14 Thread 0x7f50d6efd700 (LWP 10464) "rpc reactor-104" 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
15 Thread 0x7f50d66fc700 (LWP 10465) "rpc reactor-104" 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
16 Thread 0x7f50d5efb700 (LWP 10467) "rpc reactor-104" 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
17 Thread 0x7f50d56fa700 (LWP 10469) "client-negotiat" 0x00007f50ec82bfb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
Thread 17 (Thread 0x7f50d56fa700 (LWP 10469)):
#0 0x00007f50ec82bfb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f50ec00321c in kudu::ConditionVariable::WaitFor (this=0x7f50d56f9690, delta=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/condition_variable.cc:121
#2 0x00007f50ec12e23d in kudu::ThreadPool::DispatchThread (this=0x563cb3e76d80) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/threadpool.cc:677
#3 0x00007f50ec12f08f in kudu::ThreadPool::<lambda()>::operator()(void) const (__closure=0x563cb3ede058) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/threadpool.cc:815
#4 0x00007f50ec1307db in std::_Function_handler<void(), kudu::ThreadPool::CreateThread()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#5 0x0000563c76be735a in std::function<void ()>::operator()() const (this=0x563cb3ede058) at /usr/include/c++/7/bits/std_function.h:706
#6 0x00007f50ec11eb52 in kudu::Thread::SuperviseThread (arg=0x563cb3ede000) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#7 0x00007f50ec8256db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#8 0x00007f50ea71661f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 16 (Thread 0x7f50d5efb700 (LWP 10467)):
#0 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f50e60a025d in epoll_poll (loop=0x563cb3ed2c00, timeout=0.099383998895063996) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f50e60a4ba3 in ev_run (loop=0x563cb3ed2c00, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
W20260809 20:13:08.632596 3628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260809 20:13:08.633644 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
#3 0x00007f50ec11780f in ev::loop_ref::run (this=0x563cb4baf298, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f50e81d60f1 in kudu::rpc::ReactorThread::RunThread (this=0x563cb4baf290) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f50e81d2bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x563cb3d3ab98) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f50e81da8d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x0000563c76be735a in std::function<void ()>::operator()() const (this=0x563cb3d3ab98) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f50ec11eb52 in kudu::Thread::SuperviseThread (arg=0x563cb3d3ab40) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f50ec8256db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f50ea71661f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 15 (Thread 0x7f50d66fc700 (LWP 10465)):
#0 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f50e60a025d in epoll_poll (loop=0x563cb3e19700, timeout=0.09948251280002296) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f50e60a4ba3 in ev_run (loop=0x563cb3e19700, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f50ec11780f in ev::loop_ref::run (this=0x563cb4bae098, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f50e81d60f1 in kudu::rpc::ReactorThread::RunThread (this=0x563cb4bae090) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f50e81d2bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x563cb3d3b138) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f50e81da8d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x0000563c76be735a in std::function<void ()>::operator()() const (this=0x563cb3d3b138) at /usr/include/c++/7/bits/std_function.h:706
W20260809 20:13:08.787381 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
#8 0x00007f50ec11eb52 in kudu::Thread::SuperviseThread (arg=0x563cb3d3b0e0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f50ec8256db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f50ea71661f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 14 (Thread 0x7f50d6efd700 (LWP 10464)):
#0 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f50e60a025d in epoll_poll (loop=0x563cb3e18c00, timeout=0.099364626919850707) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f50e60a4ba3 in ev_run (loop=0x563cb3e18c00, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f50ec11780f in ev::loop_ref::run (this=0x563cb4baec98, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f50e81d60f1 in kudu::rpc::ReactorThread::RunThread (this=0x563cb4baec90) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f50e81d2bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x563cb3c48058) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f50e81da8d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x0000563c76be735a in std::function<void ()>::operator()() const (this=0x563cb3c48058) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f50ec11eb52 in kudu::Thread::SuperviseThread (arg=0x563cb3c48000) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f50ec8256db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f50ea71661f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 13 (Thread 0x7f50d7eff700 (LWP 10463)):
#0 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f50e60a025d in epoll_poll (loop=0x563cb3ed1600, timeout=0.099887462798506021) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f50e60a4ba3 in ev_run (loop=0x563cb3ed1600, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f50ec11780f in ev::loop_ref::run (this=0x563cb3e9d298, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f50e81d60f1 in kudu::rpc::ReactorThread::RunThread (this=0x563cb3e9d290) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f50e81d2bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x563cb3d3add8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f50e81da8d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x0000563c76be735a in std::function<void ()>::operator()() const (this=0x563cb3d3add8) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f50ec11eb52 in kudu::Thread::SuperviseThread (arg=0x563cb3d3ad80) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f50ec8256db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f50ea71661f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 12 (Thread 0x7f50d36f6700 (LWP 10458)):
#0 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f50e60a025d in epoll_poll (loop=0x563cb3ed2100, timeout=0.099753331625834107) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f50e60a4ba3 in ev_run (loop=0x563cb3ed2100, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f50ec11780f in ev::loop_ref::run (this=0x563cb3e9dc98, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f50e81d60f1 in kudu::rpc::ReactorThread::RunThread (this=0x563cb3e9dc90) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f50e81d2bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x563cb3e58838) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f50e81da8d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x0000563c76be735a in std::function<void ()>::operator()() const (this=0x563cb3e58838) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f50ec11eb52 in kudu::Thread::SuperviseThread (arg=0x563cb3e587e0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f50ec8256db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f50ea71661f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 11 (Thread 0x7f50d26f4700 (LWP 10457)):
#0 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f50e60a025d in epoll_poll (loop=0x563cb3ed1b80, timeout=0.09970110678113997) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f50e60a4ba3 in ev_run (loop=0x563cb3ed1b80, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f50ec11780f in ev::loop_ref::run (this=0x563cb3e9c298, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f50e81d60f1 in kudu::rpc::ReactorThread::RunThread (this=0x563cb3e9c290) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f50e81d2bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x563cb3e58958) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f50e81da8d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x0000563c76be735a in std::function<void ()>::operator()() const (this=0x563cb3e58958) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f50ec11eb52 in kudu::Thread::SuperviseThread (arg=0x563cb3e58900) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f50ec8256db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f50ea71661f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 10 (Thread 0x7f50d2ef5700 (LWP 10456)):
#0 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f50e60a025d in epoll_poll (loop=0x563cb3ed1080, timeout=0.099689177703112364) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f50e60a4ba3 in ev_run (loop=0x563cb3ed1080, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f50ec11780f in ev::loop_ref::run (this=0x563cb3e9c698, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f50e81d60f1 in kudu::rpc::ReactorThread::RunThread (this=0x563cb3e9c690) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f50e81d2bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x563cb3e58298) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f50e81da8d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x0000563c76be735a in std::function<void ()>::operator()() const (this=0x563cb3e58298) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f50ec11eb52 in kudu::Thread::SuperviseThread (arg=0x563cb3e58240) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f50ec8256db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f50ea71661f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 9 (Thread 0x7f50d76fe700 (LWP 10453)):
#0 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f50e60a025d in epoll_poll (loop=0x563cb3ed0b00, timeout=0.099630771670490503) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f50e60a4ba3 in ev_run (loop=0x563cb3ed0b00, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f50ec11780f in ev::loop_ref::run (this=0x563cb3f54498, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f50e81d60f1 in kudu::rpc::ReactorThread::RunThread (this=0x563cb3f54490) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f50e81d2bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x563cb3e59a38) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f50e81da8d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x0000563c76be735a in std::function<void ()>::operator()() const (this=0x563cb3e59a38) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f50ec11eb52 in kudu::Thread::SuperviseThread (arg=0x563cb3e599e0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f50ec8256db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f50ea71661f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 8 (Thread 0x7f50d9702700 (LWP 3969)):
#0 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f50e60a025d in epoll_poll (loop=0x563cb3e18100, timeout=0.098989046877250075) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f50e60a4ba3 in ev_run (loop=0x563cb3e18100, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f50ec11780f in ev::loop_ref::run (this=0x563cb3e30c98, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f50e81d60f1 in kudu::rpc::ReactorThread::RunThread (this=0x563cb3e30c90) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f50e81d2bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x563cb3e58ef8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f50e81da8d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x0000563c76be735a in std::function<void ()>::operator()() const (this=0x563cb3e58ef8) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f50ec11eb52 in kudu::Thread::SuperviseThread (arg=0x563cb3e58ea0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f50ec8256db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f50ea71661f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 7 (Thread 0x7f50d9f03700 (LWP 3966)):
#0 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f50e60a025d in epoll_poll (loop=0x563cb3e17b80, timeout=0.098950400250032544) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f50e60a4ba3 in ev_run (loop=0x563cb3e17b80, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f50ec11780f in ev::loop_ref::run (this=0x563cb3e30898, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f50e81d60f1 in kudu::rpc::ReactorThread::RunThread (this=0x563cb3e30890) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f50e81d2bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x563cb3e59018) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f50e81da8d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x0000563c76be735a in std::function<void ()>::operator()() const (this=0x563cb3e59018) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f50ec11eb52 in kudu::Thread::SuperviseThread (arg=0x563cb3e58fc0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f50ec8256db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f50ea71661f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 6 (Thread 0x7f50da704700 (LWP 3965)):
#0 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f50e60a025d in epoll_poll (loop=0x563cb3e16b00, timeout=0.098940430209040642) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f50e60a4ba3 in ev_run (loop=0x563cb3e16b00, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f50ec11780f in ev::loop_ref::run (this=0x563cb3e30698, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f50e81d60f1 in kudu::rpc::ReactorThread::RunThread (this=0x563cb3e30690) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f50e81d2bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x563cb3e59138) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f50e81da8d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x0000563c76be735a in std::function<void ()>::operator()() const (this=0x563cb3e59138) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f50ec11eb52 in kudu::Thread::SuperviseThread (arg=0x563cb3e590e0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f50ec8256db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f50ea71661f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 5 (Thread 0x7f50db10c700 (LWP 3963)):
#0 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f50e60a025d in epoll_poll (loop=0x563cb3e16580, timeout=0.098983277333900332) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f50e60a4ba3 in ev_run (loop=0x563cb3e16580, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f50ec11780f in ev::loop_ref::run (this=0x563cb3c67c98, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f50e81d60f1 in kudu::rpc::ReactorThread::RunThread (this=0x563cb3c67c90) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f50e81d2bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x563cb3e59258) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f50e81da8d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x0000563c76be735a in std::function<void ()>::operator()() const (this=0x563cb3e59258) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f50ec11eb52 in kudu::Thread::SuperviseThread (arg=0x563cb3e59200) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f50ec8256db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f50ea71661f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 4 (Thread 0x7f50db90d700 (LWP 3303)):
#0 0x00007f50ea716947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f50e60a025d in epoll_poll (loop=0x563cb3e16000, timeout=0.099563129711896181) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f50e60a4ba3 in ev_run (loop=0x563cb3e16000, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f50ec11780f in ev::loop_ref::run (this=0x563cb3c66e98, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f50e81d60f1 in kudu::rpc::ReactorThread::RunThread (this=0x563cb3c66e90) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f50e81d2bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x563cb3d3bd98) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f50e81da8d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x0000563c76be735a in std::function<void ()>::operator()() const (this=0x563cb3d3bd98) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f50ec11eb52 in kudu::Thread::SuperviseThread (arg=0x563cb3d3bd40) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f50ec8256db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f50ea71661f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 3 (Thread 0x7f50dc10e700 (LWP 3302)):
#0 0x00007f50ec82bfb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f50ec003033 in kudu::ConditionVariable::WaitUntil (this=0x563cb3d3bf38, until=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/condition_variable.cc:87
#2 0x00007f50ec0842f4 in kudu::CountDownLatch::WaitUntil (this=0x563cb3d3bf00, when=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/countdown_latch.h:89
#3 0x00007f50ec08439e in kudu::CountDownLatch::WaitFor (this=0x563cb3d3bf00, delta=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/countdown_latch.h:99
#4 0x00007f50ec0833b2 in kudu::KernelStackWatchdog::RunThread (this=0x563cb3d3be60) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/kernel_stack_watchdog.cc:134
#5 0x00007f50ec0825c1 in kudu::KernelStackWatchdog::<lambda()>::operator()(void) const (__closure=0x563cb3d3bb58) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/kernel_stack_watchdog.cc:71
#6 0x00007f50ec083d16 in std::_Function_handler<void(), kudu::KernelStackWatchdog::KernelStackWatchdog()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x0000563c76be735a in std::function<void ()>::operator()() const (this=0x563cb3d3bb58) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f50ec11eb52 in kudu::Thread::SuperviseThread (arg=0x563cb3d3bb00) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f50ec8256db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f50ea71661f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 2 (Thread 0x7f50ddae8700 (LWP 3301)):
#0 0x00007f50ec83032a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f50ec1153a1 in kudu::Subprocess::DoWait (this=0x7f50ddae7350, wait_status=0x0, mode=kudu::Subprocess::BLOCKING) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/subprocess.cc:849
#2 0x00007f50ec11273a in kudu::Subprocess::Wait (this=0x7f50ddae7350, wait_status=0x0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/subprocess.cc:561
#3 0x00007f50ec077729 in kudu::PstackWatcher::RunStackDump (argv=std::vector of length 12, capacity 16 = {...}) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/pstack_watcher.cc:260
#4 0x00007f50ec077259 in kudu::PstackWatcher::RunGdbStackDump (pid=3296, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/pstack_watcher.cc:218
#5 0x00007f50ec076ab7 in kudu::PstackWatcher::DumpPidStacks (pid=3296, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/pstack_watcher.cc:169
#6 0x00007f50ec076a30 in kudu::PstackWatcher::DumpStacks (flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/pstack_watcher.cc:162
#7 0x0000563c76be5e09 in kudu::<lambda()>::operator()(void) const (__closure=0x563cb3cdfff8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_main.cc:66
#8 0x0000563c76be660a in std::__invoke_impl<void, kudu::CreateAndStartTimeoutThread()::<lambda()> >(std::__invoke_other, kudu::<lambda()> &&) (__f=...) at /usr/include/c++/7/bits/invoke.h:60
#9 0x0000563c76be635e in std::__invoke<kudu::CreateAndStartTimeoutThread()::<lambda()> >(kudu::<lambda()> &&) (__fn=...) at /usr/include/c++/7/bits/invoke.h:95
#10 0x0000563c76be6d7e in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x563cb3cdfff8) at /usr/include/c++/7/thread:234
#11 0x0000563c76be6d23 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::<lambda()> > >::operator()(void) (this=0x563cb3cdfff8) at /usr/include/c++/7/thread:243
#12 0x0000563c76be6cde in std::thread::_State_impl<std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::<lambda()> > > >::_M_run(void) (this=0x563cb3cdfff0) at /usr/include/c++/7/thread:186
#13 0x00007f50eb0596df in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#14 0x00007f50ec8256db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#15 0x00007f50ea71661f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 1 (Thread 0x7f50dfb0d280 (LWP 3296)):
#0 0x00007f50ec82bad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f50ec002ead in kudu::ConditionVariable::Wait (this=0x563cb45b0eb0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/condition_variable.cc:55
W20260809 20:13:09.413833 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
#2 0x00007f50ed286aa1 in kudu::CountDownLatch::Wait (this=0x563cb45b0e78) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/countdown_latch.h:79
#3 0x00007f50ed045414 in kudu::Synchronizer::Wait (this=0x7ffc58596160) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/async_util.h:58
#4 0x00007f50ece17c0a in kudu::client::KuduClient::Data::ConnectToCluster (this=0x563cb3eab680, client=0x563cb3e8bde0, deadline=..., creds_policy=kudu::rpc::CredentialsPolicy::ANY_CREDENTIALS) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/client/client-internal.cc:828
W20260809 20:13:11.143769 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
W20260809 20:13:11.250592 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
#5 0x00007f50ecdd98a5 in kudu::client::KuduClientBuilder::Build (this=0x7ffc585965a0, client=0x563cb3e59790) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/client/client.cc:474
W20260809 20:13:11.797750 3802 consensus_peers.cc:597] T 8375821116104cf999ae49469e06def0 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1796: this message will repeat every 5th retry.
#6 0x00007f50e5482a93 in kudu::tools::CreateKuduClient (master_addresses=std::vector of length 1, capacity 1 = {...}, client=0x563cb3e59790, can_see_all_replicas=true) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/tool_action_common.cc:773
#7 0x00007f50ea502ac4 in kudu::tools::RemoteKsckCluster::Connect (this=0x563cb3e59680) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/ksck_remote.cc:530
#8 0x00007f50ea4ba88e in kudu::tools::Ksck::CheckClusterRunning (this=0x563cb3ddc010) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/ksck.cc:459
#9 0x00007f50ea4bbe84 in kudu::tools::Ksck::Run (this=0x563cb3ddc010) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/ksck.cc:594
W20260809 20:13:12.572983 3802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111) [suppressed 20 similar messages]
#10 0x00007f50ea4be27b in kudu::tools::Ksck::RunAndPrintResults (this=0x563cb3ddc010) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/ksck.cc:800
#11 0x00007f50ed257044 in kudu::ClusterVerifier::RunKsck (this=0x7ffc58596ab0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/cluster_verifier.cc:113
W20260809 20:13:13.115659 3960 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (781 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
#12 0x00007f50ed256a03 in kudu::ClusterVerifier::CheckCluster (this=0x7ffc58596ab0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/cluster_verifier.cc:78
#13 0x0000563c76b994c5 in kudu::client::AllTypesItest<kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)7> > >::RunTest(std::function<kudu::Status (kudu::client::KuduScanner*)> const&, std::function<void (kudu::client::KuduScanBatch const&, int, int*)> const&) (this=0x563cb3c33900, scanner_setup=..., verifier=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/all_types-itest.cc:622
W20260809 20:13:13.520680 3802 consensus_peers.cc:597] T 97d3cd24f4b24b578165968c3aa82513 P badc9e1d2ce547869d5d9ebf74a45b74 -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1796: this message will repeat every 5th retry.
W20260809 20:13:13.534621 3628 consensus_peers.cc:597] T 5814361c2f24496fa296e86169236f87 P 20da41f8404d40e59d12b8d87f881a1d -> Peer 58b4bccc13254c8988cf60c89783f89a (127.3.56.1:37591): Couldn't send request to peer 58b4bccc13254c8988cf60c89783f89a. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.1:37591: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
#14 0x0000563c76b6f2c1 in kudu::client::AllTypesItest_TestTimestampPadding_Test<kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)7> > >::TestBody (this=0x563cb3c33900) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/all_types-itest.cc:804
#15 0x00007f50ec2500ed in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (location=0x7f50ec25182b "the test body", method=<optimized out>, object=0x563cb3c33900) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599
#16 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0x563cb3c33900, method=<optimized out>, location=location@entry=0x7f50ec25182b "the test body") at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#17 0x00007f50ec244bdb in testing::Test::Run (this=0x563cb3c33900) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2674
#18 0x00007f50ec244d9d in testing::TestInfo::Run (this=0x563cb3c49e60) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853
#19 0x00007f50ec245377 in testing::TestSuite::Run (this=0x563cb3c48a20) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012
#20 0x00007f50ec24577c in testing::internal::UnitTestImpl::RunAllTests (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870
#21 0x00007f50ec25060d in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x7f50ec2533c8 "auxiliary test code (environments or event listeners)", method=<optimized out>, object=0x563cb3d0a280) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599
#22 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x563cb3d0a280, method=<optimized out>, location=location@entry=0x7f50ec2533c8 "auxiliary test code (environments or event listeners)") at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#23 0x00007f50ec244e63 in testing::UnitTest::Run (this=0x7f50ec4641e0 <testing::UnitTest::GetInstance()::instance>) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444
#24 0x0000563c76be70d2 in RUN_ALL_TESTS () at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/gtest/gtest.h:2293
#25 0x0000563c76be62bf in main (argc=1, argv=0x7ffc585970b8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_main.cc:115
W20260809 20:13:13.844419 3775 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:33635 (781 consecutive failures): Network error: Failed to ping master at 127.3.56.62:33635: Client connection negotiation failed: client connection to 127.3.56.62:33635: connect: Connection refused (error 111)
************************* END STACKS ***************************
F20260809 20:13:13.862625 3301 test_main.cc:69] Maximum unit test time exceeded (870 sec)
*** Check failure stack trace: ***
*** Aborted at 1786306393 (unix time) try "date -d @1786306393" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800000ce0) received by PID 3296 (TID 0x7f50ddae8700) from PID 3296; stack trace: ***
@ 0x7f50ec830980 (unknown) at ??:0
@ 0x7f50ea633e87 gsignal at ??:0
@ 0x7f50ea6357f1 abort at ??:0
@ 0x7f50eb338d5d google::LogMessage::Fail() at ??:0
@ 0x7f50eb33cb23 google::LogMessage::SendToLog() at ??:0
@ 0x7f50eb33875c google::LogMessage::Flush() at ??:0
@ 0x7f50eb339ee9 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x563c76be5f46 _ZZN4kuduL27CreateAndStartTimeoutThreadEvENKUlvE_clEv at ??:0
@ 0x563c76be660a _ZSt13__invoke_implIvZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_JEET_St14__invoke_otherOT0_DpOT1_ at ??:0
@ 0x563c76be635e _ZSt8__invokeIZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS3_DpOS4_ at ??:0
@ 0x563c76be6d7e _ZNSt6thread8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_EEE9_M_invokeIJLm0EEEEDTcl8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at ??:0
@ 0x563c76be6d23 _ZNSt6thread8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_EEEclEv at ??:0
@ 0x563c76be6cde _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_EEEEE6_M_runEv at ??:0
@ 0x7f50eb0596df (unknown) at ??:0
@ 0x7f50ec8256db start_thread at ??:0
@ 0x7f50ea71661f clone at ??:0