Note: This is test shard 3 of 8.
[==========] Running 3 tests from 3 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AllTypesItest/1, where TypeParam = kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)3> >
[ RUN ] AllTypesItest/1.TestAllKeyTypes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260810 18:34:35.656632 25920 test_util.cc:268] Using random seed: -1691899554
2026-08-10T18:34:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:34:35Z Disabled control of system clock
I20260810 18:34:35.706614 25920 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJBH3sd/build/asan/bin/kudu
/tmp/dist-test-taskJBH3sd/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.80.62:42753
--webserver_interface=127.25.80.62
--webserver_port=0
--builtin_ntp_servers=127.25.80.20:40237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.80.62:42753 with env {}
W20260810 18:34:36.340252 25929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:34:36.340932 25929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:34:36.341194 25929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:34:36.350970 25929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:34:36.351240 25929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:34:36.351402 25929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:34:36.351591 25929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:34:36.363802 25929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.80.20:40237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.80.62:42753
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.80.62:42753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.80.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 17:48:56 UTC on ccycloud.quasar-rratoj.root.comops.site
build id 4578680
ASAN enabled
I20260810 18:34:36.366142 25929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:34:36.368952 25929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:34:36.394043 25935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:34:36.393847 25940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:34:36.395210 25938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:34:36.396145 25929 server_base.cc:1034] running on GCE node
I20260810 18:34:36.397501 25929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:34:36.400318 25929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:34:36.401731 25929 hybrid_clock.cc:648] HybridClock initialized: now 1786386876401477 us; error 165 us; skew 500 ppm
I20260810 18:34:36.402374 25929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:34:36.413182 25929 webserver.cc:460] Webserver started at http://127.25.80.62:42209/ using document root <none> and password file <none>
I20260810 18:34:36.414536 25929 fs_manager.cc:359] Metadata directory not provided
I20260810 18:34:36.414752 25929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:34:36.415337 25929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:34:36.419799 25929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/instance:
uuid: "7fa3c861edac43bfb72fad37220c898f"
format_stamp: "Formatted at 2026-08-10 18:34:36 on dist-test-slave-bw88"
I20260810 18:34:36.420794 25929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/wal/instance:
uuid: "7fa3c861edac43bfb72fad37220c898f"
format_stamp: "Formatted at 2026-08-10 18:34:36 on dist-test-slave-bw88"
I20260810 18:34:36.430042 25929 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.005s
I20260810 18:34:36.435883 25949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:34:36.437983 25929 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260810 18:34:36.438275 25929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data,/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/wal
uuid: "7fa3c861edac43bfb72fad37220c898f"
format_stamp: "Formatted at 2026-08-10 18:34:36 on dist-test-slave-bw88"
I20260810 18:34:36.438493 25929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:34:36.471954 25929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:34:36.473147 25929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:34:36.473615 25929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:34:36.506841 25929 rpc_server.cc:307] RPC server started. Bound to: 127.25.80.62:42753
I20260810 18:34:36.507125 26021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.80.62:42753 every 8 connection(s)
I20260810 18:34:36.509181 25929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb
I20260810 18:34:36.514685 25920 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJBH3sd/build/asan/bin/kudu as pid 25929
I20260810 18:34:36.515318 25920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/wal/instance
I20260810 18:34:36.522701 26023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:34:36.546062 26023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fa3c861edac43bfb72fad37220c898f: Bootstrap starting.
I20260810 18:34:36.556875 26023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fa3c861edac43bfb72fad37220c898f: Neither blocks nor log segments found. Creating new log.
I20260810 18:34:36.559787 26023 log.cc:824] T 00000000000000000000000000000000 P 7fa3c861edac43bfb72fad37220c898f: Log is configured to *not* fsync() on all Append() calls
I20260810 18:34:36.566221 26023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fa3c861edac43bfb72fad37220c898f: No bootstrap required, opened a new log
I20260810 18:34:36.586440 26023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fa3c861edac43bfb72fad37220c898f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa3c861edac43bfb72fad37220c898f" member_type: VOTER last_known_addr { host: "127.25.80.62" port: 42753 } }
I20260810 18:34:36.587234 26023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fa3c861edac43bfb72fad37220c898f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:34:36.587539 26023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fa3c861edac43bfb72fad37220c898f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3c861edac43bfb72fad37220c898f, State: Initialized, Role: FOLLOWER
I20260810 18:34:36.589176 26023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fa3c861edac43bfb72fad37220c898f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa3c861edac43bfb72fad37220c898f" member_type: VOTER last_known_addr { host: "127.25.80.62" port: 42753 } }
I20260810 18:34:36.589748 26023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fa3c861edac43bfb72fad37220c898f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:34:36.590220 26023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fa3c861edac43bfb72fad37220c898f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:34:36.590632 26023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fa3c861edac43bfb72fad37220c898f [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:34:36.594871 26023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fa3c861edac43bfb72fad37220c898f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa3c861edac43bfb72fad37220c898f" member_type: VOTER last_known_addr { host: "127.25.80.62" port: 42753 } }
I20260810 18:34:36.596001 26023 leader_election.cc:302] T 00000000000000000000000000000000 P 7fa3c861edac43bfb72fad37220c898f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fa3c861edac43bfb72fad37220c898f; no voters:
I20260810 18:34:36.597483 26023 leader_election.cc:288] T 00000000000000000000000000000000 P 7fa3c861edac43bfb72fad37220c898f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:34:36.597756 26030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fa3c861edac43bfb72fad37220c898f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:34:36.600226 26030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fa3c861edac43bfb72fad37220c898f [term 1 LEADER]: Becoming Leader. State: Replica: 7fa3c861edac43bfb72fad37220c898f, State: Running, Role: LEADER
I20260810 18:34:36.601567 26030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fa3c861edac43bfb72fad37220c898f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa3c861edac43bfb72fad37220c898f" member_type: VOTER last_known_addr { host: "127.25.80.62" port: 42753 } }
I20260810 18:34:36.602414 26023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fa3c861edac43bfb72fad37220c898f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:34:36.609395 26031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fa3c861edac43bfb72fad37220c898f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fa3c861edac43bfb72fad37220c898f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa3c861edac43bfb72fad37220c898f" member_type: VOTER last_known_addr { host: "127.25.80.62" port: 42753 } } }
I20260810 18:34:36.610807 26032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fa3c861edac43bfb72fad37220c898f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fa3c861edac43bfb72fad37220c898f. Latest consensus state: current_term: 1 leader_uuid: "7fa3c861edac43bfb72fad37220c898f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa3c861edac43bfb72fad37220c898f" member_type: VOTER last_known_addr { host: "127.25.80.62" port: 42753 } } }
I20260810 18:34:36.611480 26031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fa3c861edac43bfb72fad37220c898f [sys.catalog]: This master's current role is: LEADER
I20260810 18:34:36.611603 26032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fa3c861edac43bfb72fad37220c898f [sys.catalog]: This master's current role is: LEADER
I20260810 18:34:36.622463 26039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:34:36.634418 26039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:34:36.654242 26039 catalog_manager.cc:1347] Generated new cluster ID: 9d52125ebea14ad2a53aab00fee47178
I20260810 18:34:36.654439 26039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:34:36.678766 26039 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:34:36.680557 26039 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:34:36.689868 26039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fa3c861edac43bfb72fad37220c898f: Generated new TSK 0
I20260810 18:34:36.691516 26039 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:34:36.713392 25920 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJBH3sd/build/asan/bin/kudu
/tmp/dist-test-taskJBH3sd/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.80.1:0
--local_ip_for_outbound_sockets=127.25.80.1
--webserver_interface=127.25.80.1
--webserver_port=0
--tserver_master_addrs=127.25.80.62:42753
--builtin_ntp_servers=127.25.80.20:40237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
W20260810 18:34:37.512171 26057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:34:37.512713 26057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:34:37.513028 26057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:34:37.525009 26057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:34:37.525327 26057 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260810 18:34:37.525527 26057 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260810 18:34:37.525828 26057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.80.1
I20260810 18:34:37.540197 26057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.80.20:40237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.80.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.80.1
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.25.80.62:42753
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.80.1
--log_dir=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 17:48:56 UTC on ccycloud.quasar-rratoj.root.comops.site
build id 4578680
ASAN enabled
I20260810 18:34:37.542069 26057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:34:37.544808 26057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:34:37.567414 26063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:34:37.567997 26066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:34:37.568415 26064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:34:37.569977 26057 server_base.cc:1034] running on GCE node
I20260810 18:34:37.570691 26057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:34:37.571934 26057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:34:37.573304 26057 hybrid_clock.cc:648] HybridClock initialized: now 1786386877573114 us; error 61 us; skew 500 ppm
I20260810 18:34:37.573792 26057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:34:37.578147 26057 webserver.cc:460] Webserver started at http://127.25.80.1:46499/ using document root <none> and password file <none>
I20260810 18:34:37.579092 26057 fs_manager.cc:359] Metadata directory not provided
I20260810 18:34:37.579347 26057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:34:37.579836 26057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:34:37.583611 26057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/instance:
uuid: "31cbb92d24564804aebba44e82409852"
format_stamp: "Formatted at 2026-08-10 18:34:37 on dist-test-slave-bw88"
I20260810 18:34:37.584731 26057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/wal/instance:
uuid: "31cbb92d24564804aebba44e82409852"
format_stamp: "Formatted at 2026-08-10 18:34:37 on dist-test-slave-bw88"
I20260810 18:34:37.593130 26057 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.001s
I20260810 18:34:37.598598 26075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:34:37.600354 26057 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:34:37.600564 26057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/wal
uuid: "31cbb92d24564804aebba44e82409852"
format_stamp: "Formatted at 2026-08-10 18:34:37 on dist-test-slave-bw88"
I20260810 18:34:37.600857 26057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:34:37.622032 26057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:34:37.623498 26057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:34:37.624044 26057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:34:37.626052 26057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:34:37.628739 26057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:34:37.628880 26057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:34:37.629148 26057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:34:37.629249 26057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:34:37.741171 26057 rpc_server.cc:307] RPC server started. Bound to: 127.25.80.1:41319
I20260810 18:34:37.741369 26230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.80.1:41319 every 8 connection(s)
I20260810 18:34:37.743301 26057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb
I20260810 18:34:37.752851 25920 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJBH3sd/build/asan/bin/kudu as pid 26057
I20260810 18:34:37.753167 25920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/wal/instance
I20260810 18:34:37.758508 25920 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJBH3sd/build/asan/bin/kudu
/tmp/dist-test-taskJBH3sd/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.80.2:0
--local_ip_for_outbound_sockets=127.25.80.2
--webserver_interface=127.25.80.2
--webserver_port=0
--tserver_master_addrs=127.25.80.62:42753
--builtin_ntp_servers=127.25.80.20:40237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
I20260810 18:34:37.772130 26231 heartbeater.cc:347] Connected to a master server at 127.25.80.62:42753
I20260810 18:34:37.772686 26231 heartbeater.cc:464] Registering TS with master...
I20260810 18:34:37.774464 26231 heartbeater.cc:511] Master 127.25.80.62:42753 requested a full tablet report, sending...
I20260810 18:34:37.779148 25972 ts_manager.cc:194] Registered new tserver with Master: 31cbb92d24564804aebba44e82409852 (127.25.80.1:41319)
I20260810 18:34:37.783186 25972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.80.1:55325
W20260810 18:34:38.516352 26235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:34:38.516806 26235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:34:38.517017 26235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:34:38.528134 26235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:34:38.528352 26235 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260810 18:34:38.528455 26235 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260810 18:34:38.528672 26235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.80.2
I20260810 18:34:38.541507 26235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.80.20:40237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.80.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.80.2
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.25.80.62:42753
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.80.2
--log_dir=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 17:48:56 UTC on ccycloud.quasar-rratoj.root.comops.site
build id 4578680
ASAN enabled
I20260810 18:34:38.543140 26235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:34:38.545535 26235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:34:38.568082 26242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:34:38.568487 26243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:34:38.568585 26246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:34:38.570834 26235 server_base.cc:1034] running on GCE node
I20260810 18:34:38.571791 26235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:34:38.573057 26235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:34:38.574486 26235 hybrid_clock.cc:648] HybridClock initialized: now 1786386878574254 us; error 58 us; skew 500 ppm
I20260810 18:34:38.574985 26235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:34:38.579276 26235 webserver.cc:460] Webserver started at http://127.25.80.2:32971/ using document root <none> and password file <none>
I20260810 18:34:38.580241 26235 fs_manager.cc:359] Metadata directory not provided
I20260810 18:34:38.580503 26235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:34:38.580981 26235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:34:38.584591 26235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-1/data/instance:
uuid: "878235fb05a24dd48d48525ad7dbf3a5"
format_stamp: "Formatted at 2026-08-10 18:34:38 on dist-test-slave-bw88"
I20260810 18:34:38.586012 26235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-1/wal/instance:
uuid: "878235fb05a24dd48d48525ad7dbf3a5"
format_stamp: "Formatted at 2026-08-10 18:34:38 on dist-test-slave-bw88"
I20260810 18:34:38.594302 26235 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.002s
I20260810 18:34:38.599474 26254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:34:38.601084 26235 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260810 18:34:38.601418 26235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-1/wal
uuid: "878235fb05a24dd48d48525ad7dbf3a5"
format_stamp: "Formatted at 2026-08-10 18:34:38 on dist-test-slave-bw88"
I20260810 18:34:38.601747 26235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:34:38.626956 26235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:34:38.628484 26235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:34:38.629000 26235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:34:38.630926 26235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:34:38.633526 26235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:34:38.633682 26235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:34:38.633960 26235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:34:38.634104 26235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:34:38.732366 26235 rpc_server.cc:307] RPC server started. Bound to: 127.25.80.2:44027
I20260810 18:34:38.732411 26430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.80.2:44027 every 8 connection(s)
I20260810 18:34:38.734289 26235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-1/data/info.pb
I20260810 18:34:38.740020 25920 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJBH3sd/build/asan/bin/kudu as pid 26235
I20260810 18:34:38.740392 25920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-1/wal/instance
I20260810 18:34:38.744434 25920 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJBH3sd/build/asan/bin/kudu
/tmp/dist-test-taskJBH3sd/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.80.3:0
--local_ip_for_outbound_sockets=127.25.80.3
--webserver_interface=127.25.80.3
--webserver_port=0
--tserver_master_addrs=127.25.80.62:42753
--builtin_ntp_servers=127.25.80.20:40237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
I20260810 18:34:38.760213 26433 heartbeater.cc:347] Connected to a master server at 127.25.80.62:42753
I20260810 18:34:38.760654 26433 heartbeater.cc:464] Registering TS with master...
I20260810 18:34:38.761960 26433 heartbeater.cc:511] Master 127.25.80.62:42753 requested a full tablet report, sending...
I20260810 18:34:38.764065 25972 ts_manager.cc:194] Registered new tserver with Master: 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027)
I20260810 18:34:38.765651 25972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.80.2:43045
I20260810 18:34:38.788367 26231 heartbeater.cc:503] Master 127.25.80.62:42753 was elected leader, sending a full tablet report...
W20260810 18:34:39.492762 26444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:34:39.493207 26444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:34:39.493552 26444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:34:39.505455 26444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:34:39.505730 26444 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260810 18:34:39.505831 26444 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260810 18:34:39.506155 26444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.80.3
I20260810 18:34:39.524631 26444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.80.20:40237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.80.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.80.3
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.25.80.62:42753
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.80.3
--log_dir=/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 17:48:56 UTC on ccycloud.quasar-rratoj.root.comops.site
build id 4578680
ASAN enabled
I20260810 18:34:39.526405 26444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:34:39.528594 26444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:34:39.550058 26450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:34:39.550271 26451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:34:39.550699 26456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:34:39.552454 26444 server_base.cc:1034] running on GCE node
I20260810 18:34:39.553277 26444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:34:39.554440 26444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:34:39.555794 26444 hybrid_clock.cc:648] HybridClock initialized: now 1786386879555607 us; error 42 us; skew 500 ppm
I20260810 18:34:39.556226 26444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:34:39.560439 26444 webserver.cc:460] Webserver started at http://127.25.80.3:41587/ using document root <none> and password file <none>
I20260810 18:34:39.561379 26444 fs_manager.cc:359] Metadata directory not provided
I20260810 18:34:39.561617 26444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:34:39.562042 26444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:34:39.565461 26444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/instance:
uuid: "5f58fd4898774f4db897e97cc1598adf"
format_stamp: "Formatted at 2026-08-10 18:34:39 on dist-test-slave-bw88"
I20260810 18:34:39.566474 26444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/wal/instance:
uuid: "5f58fd4898774f4db897e97cc1598adf"
format_stamp: "Formatted at 2026-08-10 18:34:39 on dist-test-slave-bw88"
I20260810 18:34:39.573577 26444 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260810 18:34:39.578328 26468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:34:39.579991 26444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 18:34:39.580228 26444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/wal
uuid: "5f58fd4898774f4db897e97cc1598adf"
format_stamp: "Formatted at 2026-08-10 18:34:39 on dist-test-slave-bw88"
I20260810 18:34:39.580528 26444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:34:39.606729 26444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:34:39.608170 26444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:34:39.608788 26444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:34:39.610914 26444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:34:39.613644 26444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:34:39.613795 26444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:34:39.614049 26444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:34:39.614224 26444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:34:39.726496 26444 rpc_server.cc:307] RPC server started. Bound to: 127.25.80.3:40545
I20260810 18:34:39.726755 26621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.80.3:40545 every 8 connection(s)
I20260810 18:34:39.728300 26444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb
I20260810 18:34:39.738579 25920 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJBH3sd/build/asan/bin/kudu as pid 26444
I20260810 18:34:39.738832 25920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/wal/instance
I20260810 18:34:39.756857 26623 heartbeater.cc:347] Connected to a master server at 127.25.80.62:42753
I20260810 18:34:39.757288 26623 heartbeater.cc:464] Registering TS with master...
I20260810 18:34:39.758601 26623 heartbeater.cc:511] Master 127.25.80.62:42753 requested a full tablet report, sending...
I20260810 18:34:39.760609 25972 ts_manager.cc:194] Registered new tserver with Master: 5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545)
I20260810 18:34:39.761736 25972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.80.3:42409
I20260810 18:34:39.768955 26433 heartbeater.cc:503] Master 127.25.80.62:42753 was elected leader, sending a full tablet report...
I20260810 18:34:39.771181 25920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:34:39.803153 25970 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60800:
name: "all-types-table"
schema {
columns {
name: "key"
type: INT16
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int8_val"
type: INT8
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int16_val"
type: INT16
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int32_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int64_val"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "timestamp_val"
type: UNIXTIME_MICROS
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "date_val"
type: DATE
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "varchar_val"
type: VARCHAR
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
length: 65535
}
immutable: false
}
columns {
name: "bool_val"
type: BOOL
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "float_val"
type: FLOAT
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "double_val"
type: DOUBLE
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "binary_val"
type: BINARY
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "decimal32_val"
type: DECIMAL32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 9
scale: 0
}
immutable: false
}
columns {
name: "decimal64_val"
type: DECIMAL64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 18
scale: 0
}
immutable: false
}
columns {
name: "decimal128_val"
type: DECIMAL128
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 38
scale: 0
}
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\000\000\252*\004\001\000\000\000TU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:34:39.817183 25970 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table all-types-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:34:39.890945 26139 tablet_service.cc:1467] Processing CreateTablet for tablet 58bd2e2d08644a7883215728907d24b4 (DEFAULT_TABLE table=all-types-table [id=7be9ece7afc9434bb89f5f7e2c2d1e32]), partition=RANGE (key) PARTITION 21844 <= VALUES
I20260810 18:34:39.891418 26331 tablet_service.cc:1467] Processing CreateTablet for tablet 58bd2e2d08644a7883215728907d24b4 (DEFAULT_TABLE table=all-types-table [id=7be9ece7afc9434bb89f5f7e2c2d1e32]), partition=RANGE (key) PARTITION 21844 <= VALUES
I20260810 18:34:39.892432 26334 tablet_service.cc:1467] Processing CreateTablet for tablet cfeba2dcf4954505b96a807ac76f55ce (DEFAULT_TABLE table=all-types-table [id=7be9ece7afc9434bb89f5f7e2c2d1e32]), partition=RANGE (key) PARTITION VALUES < 10922
I20260810 18:34:39.893280 26332 tablet_service.cc:1467] Processing CreateTablet for tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 (DEFAULT_TABLE table=all-types-table [id=7be9ece7afc9434bb89f5f7e2c2d1e32]), partition=RANGE (key) PARTITION 10922 <= VALUES < 21844
I20260810 18:34:39.892711 26141 tablet_service.cc:1467] Processing CreateTablet for tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 (DEFAULT_TABLE table=all-types-table [id=7be9ece7afc9434bb89f5f7e2c2d1e32]), partition=RANGE (key) PARTITION 10922 <= VALUES < 21844
I20260810 18:34:39.896423 26141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35ffbeab79c14b74b8d98a6fad1fbfa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:34:39.896917 26331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58bd2e2d08644a7883215728907d24b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:34:39.899430 26538 tablet_service.cc:1467] Processing CreateTablet for tablet cfeba2dcf4954505b96a807ac76f55ce (DEFAULT_TABLE table=all-types-table [id=7be9ece7afc9434bb89f5f7e2c2d1e32]), partition=RANGE (key) PARTITION VALUES < 10922
I20260810 18:34:39.902135 26332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35ffbeab79c14b74b8d98a6fad1fbfa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:34:39.901675 26535 tablet_service.cc:1467] Processing CreateTablet for tablet 58bd2e2d08644a7883215728907d24b4 (DEFAULT_TABLE table=all-types-table [id=7be9ece7afc9434bb89f5f7e2c2d1e32]), partition=RANGE (key) PARTITION 21844 <= VALUES
I20260810 18:34:39.902740 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfeba2dcf4954505b96a807ac76f55ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:34:39.898633 26537 tablet_service.cc:1467] Processing CreateTablet for tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 (DEFAULT_TABLE table=all-types-table [id=7be9ece7afc9434bb89f5f7e2c2d1e32]), partition=RANGE (key) PARTITION 10922 <= VALUES < 21844
I20260810 18:34:39.903545 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35ffbeab79c14b74b8d98a6fad1fbfa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:34:39.904366 26535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58bd2e2d08644a7883215728907d24b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:34:39.904716 26139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58bd2e2d08644a7883215728907d24b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:34:39.906641 26334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfeba2dcf4954505b96a807ac76f55ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:34:39.906939 26143 tablet_service.cc:1467] Processing CreateTablet for tablet cfeba2dcf4954505b96a807ac76f55ce (DEFAULT_TABLE table=all-types-table [id=7be9ece7afc9434bb89f5f7e2c2d1e32]), partition=RANGE (key) PARTITION VALUES < 10922
I20260810 18:34:39.907840 26143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfeba2dcf4954505b96a807ac76f55ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:34:39.932204 26649 tablet_bootstrap.cc:492] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5: Bootstrap starting.
I20260810 18:34:39.940609 26649 tablet_bootstrap.cc:654] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5: Neither blocks nor log segments found. Creating new log.
I20260810 18:34:39.943368 26649 log.cc:824] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5: Log is configured to *not* fsync() on all Append() calls
I20260810 18:34:39.943949 26653 tablet_bootstrap.cc:492] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852: Bootstrap starting.
I20260810 18:34:39.946177 26650 tablet_bootstrap.cc:492] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf: Bootstrap starting.
I20260810 18:34:39.954890 26649 tablet_bootstrap.cc:492] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5: No bootstrap required, opened a new log
I20260810 18:34:39.955510 26649 ts_tablet_manager.cc:1397] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.013s
I20260810 18:34:39.956758 26650 tablet_bootstrap.cc:654] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf: Neither blocks nor log segments found. Creating new log.
I20260810 18:34:39.958724 26650 log.cc:824] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf: Log is configured to *not* fsync() on all Append() calls
I20260810 18:34:39.959975 26653 tablet_bootstrap.cc:654] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852: Neither blocks nor log segments found. Creating new log.
I20260810 18:34:39.963155 26653 log.cc:824] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852: Log is configured to *not* fsync() on all Append() calls
I20260810 18:34:39.964191 26650 tablet_bootstrap.cc:492] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf: No bootstrap required, opened a new log
I20260810 18:34:39.964747 26650 ts_tablet_manager.cc:1397] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.012s
I20260810 18:34:39.969744 26653 tablet_bootstrap.cc:492] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852: No bootstrap required, opened a new log
I20260810 18:34:39.970255 26653 ts_tablet_manager.cc:1397] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.004s
I20260810 18:34:39.975641 26649 raft_consensus.cc:348] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
I20260810 18:34:39.976616 26649 raft_consensus.cc:374] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:34:39.976820 26649 raft_consensus.cc:729] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 878235fb05a24dd48d48525ad7dbf3a5, State: Initialized, Role: FOLLOWER
I20260810 18:34:39.978044 26649 consensus_queue.cc:260] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
I20260810 18:34:39.979728 26650 raft_consensus.cc:348] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } }
I20260810 18:34:39.980499 26650 raft_consensus.cc:374] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:34:39.980731 26650 raft_consensus.cc:729] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f58fd4898774f4db897e97cc1598adf, State: Initialized, Role: FOLLOWER
I20260810 18:34:39.981884 26649 ts_tablet_manager.cc:1428] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5: Time spent starting tablet: real 0.026s user 0.017s sys 0.010s
I20260810 18:34:39.981729 26650 consensus_queue.cc:260] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } }
I20260810 18:34:39.982580 26649 tablet_bootstrap.cc:492] T cfeba2dcf4954505b96a807ac76f55ce P 878235fb05a24dd48d48525ad7dbf3a5: Bootstrap starting.
I20260810 18:34:39.986295 26649 tablet_bootstrap.cc:654] T cfeba2dcf4954505b96a807ac76f55ce P 878235fb05a24dd48d48525ad7dbf3a5: Neither blocks nor log segments found. Creating new log.
W20260810 18:34:39.988633 26437 tablet.cc:2369] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:34:39.992538 26653 raft_consensus.cc:348] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
I20260810 18:34:39.994314 26653 raft_consensus.cc:374] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:34:39.994580 26653 raft_consensus.cc:729] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31cbb92d24564804aebba44e82409852, State: Initialized, Role: FOLLOWER
I20260810 18:34:39.995958 26653 consensus_queue.cc:260] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
I20260810 18:34:39.998190 26623 heartbeater.cc:503] Master 127.25.80.62:42753 was elected leader, sending a full tablet report...
I20260810 18:34:39.998513 26649 tablet_bootstrap.cc:492] T cfeba2dcf4954505b96a807ac76f55ce P 878235fb05a24dd48d48525ad7dbf3a5: No bootstrap required, opened a new log
I20260810 18:34:39.998798 26649 ts_tablet_manager.cc:1397] T cfeba2dcf4954505b96a807ac76f55ce P 878235fb05a24dd48d48525ad7dbf3a5: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.008s
I20260810 18:34:39.999265 26650 ts_tablet_manager.cc:1428] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf: Time spent starting tablet: real 0.034s user 0.000s sys 0.034s
I20260810 18:34:40.000393 26650 tablet_bootstrap.cc:492] T 58bd2e2d08644a7883215728907d24b4 P 5f58fd4898774f4db897e97cc1598adf: Bootstrap starting.
I20260810 18:34:40.000656 26649 raft_consensus.cc:348] T cfeba2dcf4954505b96a807ac76f55ce P 878235fb05a24dd48d48525ad7dbf3a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } }
I20260810 18:34:40.001066 26649 raft_consensus.cc:374] T cfeba2dcf4954505b96a807ac76f55ce P 878235fb05a24dd48d48525ad7dbf3a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:34:40.001430 26649 raft_consensus.cc:729] T cfeba2dcf4954505b96a807ac76f55ce P 878235fb05a24dd48d48525ad7dbf3a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 878235fb05a24dd48d48525ad7dbf3a5, State: Initialized, Role: FOLLOWER
I20260810 18:34:40.001956 26649 consensus_queue.cc:260] T cfeba2dcf4954505b96a807ac76f55ce P 878235fb05a24dd48d48525ad7dbf3a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } }
I20260810 18:34:40.003135 26649 ts_tablet_manager.cc:1428] T cfeba2dcf4954505b96a807ac76f55ce P 878235fb05a24dd48d48525ad7dbf3a5: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260810 18:34:40.003614 26649 tablet_bootstrap.cc:492] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5: Bootstrap starting.
I20260810 18:34:40.004761 26650 tablet_bootstrap.cc:654] T 58bd2e2d08644a7883215728907d24b4 P 5f58fd4898774f4db897e97cc1598adf: Neither blocks nor log segments found. Creating new log.
I20260810 18:34:40.007715 26649 tablet_bootstrap.cc:654] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5: Neither blocks nor log segments found. Creating new log.
I20260810 18:34:40.013312 26653 ts_tablet_manager.cc:1428] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852: Time spent starting tablet: real 0.043s user 0.009s sys 0.030s
I20260810 18:34:40.014158 26653 tablet_bootstrap.cc:492] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 31cbb92d24564804aebba44e82409852: Bootstrap starting.
I20260810 18:34:40.015664 26650 tablet_bootstrap.cc:492] T 58bd2e2d08644a7883215728907d24b4 P 5f58fd4898774f4db897e97cc1598adf: No bootstrap required, opened a new log
I20260810 18:34:40.015950 26650 ts_tablet_manager.cc:1397] T 58bd2e2d08644a7883215728907d24b4 P 5f58fd4898774f4db897e97cc1598adf: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.011s
I20260810 18:34:40.018332 26650 raft_consensus.cc:348] T 58bd2e2d08644a7883215728907d24b4 P 5f58fd4898774f4db897e97cc1598adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
I20260810 18:34:40.018761 26650 raft_consensus.cc:374] T 58bd2e2d08644a7883215728907d24b4 P 5f58fd4898774f4db897e97cc1598adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:34:40.018998 26650 raft_consensus.cc:729] T 58bd2e2d08644a7883215728907d24b4 P 5f58fd4898774f4db897e97cc1598adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f58fd4898774f4db897e97cc1598adf, State: Initialized, Role: FOLLOWER
I20260810 18:34:40.019073 26653 tablet_bootstrap.cc:654] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 31cbb92d24564804aebba44e82409852: Neither blocks nor log segments found. Creating new log.
I20260810 18:34:40.019420 26650 consensus_queue.cc:260] T 58bd2e2d08644a7883215728907d24b4 P 5f58fd4898774f4db897e97cc1598adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
I20260810 18:34:40.019637 26649 tablet_bootstrap.cc:492] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5: No bootstrap required, opened a new log
I20260810 18:34:40.020006 26649 ts_tablet_manager.cc:1397] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.010s
I20260810 18:34:40.020210 26650 ts_tablet_manager.cc:1428] T 58bd2e2d08644a7883215728907d24b4 P 5f58fd4898774f4db897e97cc1598adf: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260810 18:34:40.020596 26650 tablet_bootstrap.cc:492] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf: Bootstrap starting.
I20260810 18:34:40.021775 26649 raft_consensus.cc:348] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
I20260810 18:34:40.022256 26649 raft_consensus.cc:374] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:34:40.022475 26649 raft_consensus.cc:729] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 878235fb05a24dd48d48525ad7dbf3a5, State: Initialized, Role: FOLLOWER
I20260810 18:34:40.023202 26649 consensus_queue.cc:260] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
I20260810 18:34:40.024263 26649 ts_tablet_manager.cc:1428] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260810 18:34:40.024421 26650 tablet_bootstrap.cc:654] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf: Neither blocks nor log segments found. Creating new log.
I20260810 18:34:40.029325 26653 tablet_bootstrap.cc:492] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 31cbb92d24564804aebba44e82409852: No bootstrap required, opened a new log
I20260810 18:34:40.029241 26659 raft_consensus.cc:482] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:34:40.029809 26653 ts_tablet_manager.cc:1397] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 31cbb92d24564804aebba44e82409852: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.010s
I20260810 18:34:40.029879 26659 raft_consensus.cc:504] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } }
I20260810 18:34:40.031633 26653 raft_consensus.cc:348] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 31cbb92d24564804aebba44e82409852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
I20260810 18:34:40.031921 26653 raft_consensus.cc:374] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 31cbb92d24564804aebba44e82409852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:34:40.032193 26653 raft_consensus.cc:729] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 31cbb92d24564804aebba44e82409852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31cbb92d24564804aebba44e82409852, State: Initialized, Role: FOLLOWER
I20260810 18:34:40.032593 26650 tablet_bootstrap.cc:492] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf: No bootstrap required, opened a new log
I20260810 18:34:40.032732 26653 consensus_queue.cc:260] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 31cbb92d24564804aebba44e82409852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
I20260810 18:34:40.033022 26650 ts_tablet_manager.cc:1397] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.001s
I20260810 18:34:40.033682 26653 ts_tablet_manager.cc:1428] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 31cbb92d24564804aebba44e82409852: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260810 18:34:40.034099 26653 tablet_bootstrap.cc:492] T cfeba2dcf4954505b96a807ac76f55ce P 31cbb92d24564804aebba44e82409852: Bootstrap starting.
I20260810 18:34:40.034760 26650 raft_consensus.cc:348] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
I20260810 18:34:40.035287 26650 raft_consensus.cc:374] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:34:40.035535 26650 raft_consensus.cc:729] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f58fd4898774f4db897e97cc1598adf, State: Initialized, Role: FOLLOWER
I20260810 18:34:40.036052 26650 consensus_queue.cc:260] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
I20260810 18:34:40.037822 26653 tablet_bootstrap.cc:654] T cfeba2dcf4954505b96a807ac76f55ce P 31cbb92d24564804aebba44e82409852: Neither blocks nor log segments found. Creating new log.
I20260810 18:34:40.041638 26659 leader_election.cc:288] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31cbb92d24564804aebba44e82409852 (127.25.80.1:41319), 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027)
I20260810 18:34:40.043381 26653 tablet_bootstrap.cc:492] T cfeba2dcf4954505b96a807ac76f55ce P 31cbb92d24564804aebba44e82409852: No bootstrap required, opened a new log
I20260810 18:34:40.043784 26653 ts_tablet_manager.cc:1397] T cfeba2dcf4954505b96a807ac76f55ce P 31cbb92d24564804aebba44e82409852: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20260810 18:34:40.045603 26653 raft_consensus.cc:348] T cfeba2dcf4954505b96a807ac76f55ce P 31cbb92d24564804aebba44e82409852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } }
I20260810 18:34:40.046159 26653 raft_consensus.cc:374] T cfeba2dcf4954505b96a807ac76f55ce P 31cbb92d24564804aebba44e82409852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:34:40.046417 26653 raft_consensus.cc:729] T cfeba2dcf4954505b96a807ac76f55ce P 31cbb92d24564804aebba44e82409852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31cbb92d24564804aebba44e82409852, State: Initialized, Role: FOLLOWER
I20260810 18:34:40.046907 26653 consensus_queue.cc:260] T cfeba2dcf4954505b96a807ac76f55ce P 31cbb92d24564804aebba44e82409852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } }
I20260810 18:34:40.047592 26658 raft_consensus.cc:482] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:34:40.048206 26658 raft_consensus.cc:504] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
I20260810 18:34:40.049468 26653 ts_tablet_manager.cc:1428] T cfeba2dcf4954505b96a807ac76f55ce P 31cbb92d24564804aebba44e82409852: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260810 18:34:40.052707 26169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfeba2dcf4954505b96a807ac76f55ce" candidate_uuid: "5f58fd4898774f4db897e97cc1598adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31cbb92d24564804aebba44e82409852" is_pre_election: true
I20260810 18:34:40.054049 26169 raft_consensus.cc:2454] T cfeba2dcf4954505b96a807ac76f55ce P 31cbb92d24564804aebba44e82409852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f58fd4898774f4db897e97cc1598adf in term 0.
I20260810 18:34:40.057603 26658 leader_election.cc:288] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31cbb92d24564804aebba44e82409852 (127.25.80.1:41319), 5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545)
I20260810 18:34:40.062657 26650 ts_tablet_manager.cc:1428] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf: Time spent starting tablet: real 0.029s user 0.000s sys 0.026s
I20260810 18:34:40.065670 26472 leader_election.cc:302] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31cbb92d24564804aebba44e82409852, 5f58fd4898774f4db897e97cc1598adf; no voters:
I20260810 18:34:40.066542 26670 raft_consensus.cc:2781] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:34:40.067975 26658 raft_consensus.cc:482] T cfeba2dcf4954505b96a807ac76f55ce P 878235fb05a24dd48d48525ad7dbf3a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:34:40.066922 26670 raft_consensus.cc:482] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:34:40.068529 26670 raft_consensus.cc:3037] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:34:40.068354 26658 raft_consensus.cc:504] T cfeba2dcf4954505b96a807ac76f55ce P 878235fb05a24dd48d48525ad7dbf3a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } }
I20260810 18:34:40.069286 26658 leader_election.cc:288] T cfeba2dcf4954505b96a807ac76f55ce P 878235fb05a24dd48d48525ad7dbf3a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31cbb92d24564804aebba44e82409852 (127.25.80.1:41319), 5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545)
I20260810 18:34:40.075074 26670 raft_consensus.cc:504] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } }
I20260810 18:34:40.076822 26670 leader_election.cc:288] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf [CANDIDATE]: Term 1 election: Requested vote from peers 31cbb92d24564804aebba44e82409852 (127.25.80.1:41319), 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027)
I20260810 18:34:40.078573 26169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfeba2dcf4954505b96a807ac76f55ce" candidate_uuid: "5f58fd4898774f4db897e97cc1598adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31cbb92d24564804aebba44e82409852"
I20260810 18:34:40.079005 26169 raft_consensus.cc:3037] T cfeba2dcf4954505b96a807ac76f55ce P 31cbb92d24564804aebba44e82409852 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:34:40.081748 26168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bd2e2d08644a7883215728907d24b4" candidate_uuid: "878235fb05a24dd48d48525ad7dbf3a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31cbb92d24564804aebba44e82409852" is_pre_election: true
I20260810 18:34:40.082161 26167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfeba2dcf4954505b96a807ac76f55ce" candidate_uuid: "878235fb05a24dd48d48525ad7dbf3a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31cbb92d24564804aebba44e82409852" is_pre_election: true
I20260810 18:34:40.082741 26168 raft_consensus.cc:2454] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 878235fb05a24dd48d48525ad7dbf3a5 in term 0.
I20260810 18:34:40.084347 26260 leader_election.cc:302] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31cbb92d24564804aebba44e82409852, 878235fb05a24dd48d48525ad7dbf3a5; no voters:
I20260810 18:34:40.085155 26658 raft_consensus.cc:2781] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:34:40.085419 26658 raft_consensus.cc:482] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:34:40.085690 26658 raft_consensus.cc:3037] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:34:40.086123 26169 raft_consensus.cc:2454] T cfeba2dcf4954505b96a807ac76f55ce P 31cbb92d24564804aebba44e82409852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f58fd4898774f4db897e97cc1598adf in term 1.
I20260810 18:34:40.086769 26472 leader_election.cc:302] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31cbb92d24564804aebba44e82409852, 5f58fd4898774f4db897e97cc1598adf; no voters:
I20260810 18:34:40.087203 26670 raft_consensus.cc:2781] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:34:40.087743 26670 raft_consensus.cc:686] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf [term 1 LEADER]: Becoming Leader. State: Replica: 5f58fd4898774f4db897e97cc1598adf, State: Running, Role: LEADER
I20260810 18:34:40.088775 26670 consensus_queue.cc:237] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } }
I20260810 18:34:40.089471 26658 raft_consensus.cc:504] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
I20260810 18:34:40.090440 26658 leader_election.cc:288] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5 [CANDIDATE]: Term 1 election: Requested vote from peers 31cbb92d24564804aebba44e82409852 (127.25.80.1:41319), 5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545)
I20260810 18:34:40.090824 26169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bd2e2d08644a7883215728907d24b4" candidate_uuid: "878235fb05a24dd48d48525ad7dbf3a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31cbb92d24564804aebba44e82409852"
I20260810 18:34:40.091166 26169 raft_consensus.cc:3037] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:34:40.094456 26169 raft_consensus.cc:2454] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 878235fb05a24dd48d48525ad7dbf3a5 in term 1.
I20260810 18:34:40.094866 26658 raft_consensus.cc:482] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:34:40.094491 26566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bd2e2d08644a7883215728907d24b4" candidate_uuid: "878235fb05a24dd48d48525ad7dbf3a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f58fd4898774f4db897e97cc1598adf"
I20260810 18:34:40.095180 26658 raft_consensus.cc:504] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
I20260810 18:34:40.095417 26566 raft_consensus.cc:3037] T 58bd2e2d08644a7883215728907d24b4 P 5f58fd4898774f4db897e97cc1598adf [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:34:40.095968 26658 leader_election.cc:288] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31cbb92d24564804aebba44e82409852 (127.25.80.1:41319), 5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545)
I20260810 18:34:40.096123 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfeba2dcf4954505b96a807ac76f55ce" candidate_uuid: "878235fb05a24dd48d48525ad7dbf3a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f58fd4898774f4db897e97cc1598adf" is_pre_election: true
I20260810 18:34:40.096379 26363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfeba2dcf4954505b96a807ac76f55ce" candidate_uuid: "5f58fd4898774f4db897e97cc1598adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "878235fb05a24dd48d48525ad7dbf3a5"
I20260810 18:34:40.096277 26364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfeba2dcf4954505b96a807ac76f55ce" candidate_uuid: "5f58fd4898774f4db897e97cc1598adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "878235fb05a24dd48d48525ad7dbf3a5" is_pre_election: true
I20260810 18:34:40.097191 26363 raft_consensus.cc:3037] T cfeba2dcf4954505b96a807ac76f55ce P 878235fb05a24dd48d48525ad7dbf3a5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:34:40.098150 26260 leader_election.cc:302] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31cbb92d24564804aebba44e82409852, 878235fb05a24dd48d48525ad7dbf3a5; no voters:
I20260810 18:34:40.094249 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bd2e2d08644a7883215728907d24b4" candidate_uuid: "878235fb05a24dd48d48525ad7dbf3a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f58fd4898774f4db897e97cc1598adf" is_pre_election: true
I20260810 18:34:40.098623 26565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ffbeab79c14b74b8d98a6fad1fbfa9" candidate_uuid: "878235fb05a24dd48d48525ad7dbf3a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f58fd4898774f4db897e97cc1598adf" is_pre_election: true
I20260810 18:34:40.099601 26565 raft_consensus.cc:2454] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 878235fb05a24dd48d48525ad7dbf3a5 in term 0.
I20260810 18:34:40.100472 26256 leader_election.cc:302] T cfeba2dcf4954505b96a807ac76f55ce P 878235fb05a24dd48d48525ad7dbf3a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 878235fb05a24dd48d48525ad7dbf3a5; no voters: 31cbb92d24564804aebba44e82409852, 5f58fd4898774f4db897e97cc1598adf
I20260810 18:34:40.103277 26363 raft_consensus.cc:2454] T cfeba2dcf4954505b96a807ac76f55ce P 878235fb05a24dd48d48525ad7dbf3a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f58fd4898774f4db897e97cc1598adf in term 1.
I20260810 18:34:40.104877 26679 raft_consensus.cc:2781] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:34:40.105244 26256 leader_election.cc:302] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f58fd4898774f4db897e97cc1598adf, 878235fb05a24dd48d48525ad7dbf3a5; no voters:
I20260810 18:34:40.109730 26566 raft_consensus.cc:2454] T 58bd2e2d08644a7883215728907d24b4 P 5f58fd4898774f4db897e97cc1598adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 878235fb05a24dd48d48525ad7dbf3a5 in term 1.
I20260810 18:34:40.112205 26169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ffbeab79c14b74b8d98a6fad1fbfa9" candidate_uuid: "878235fb05a24dd48d48525ad7dbf3a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31cbb92d24564804aebba44e82409852" is_pre_election: true
I20260810 18:34:40.112438 26658 raft_consensus.cc:2726] T cfeba2dcf4954505b96a807ac76f55ce P 878235fb05a24dd48d48525ad7dbf3a5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:34:40.112722 26169 raft_consensus.cc:2454] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 31cbb92d24564804aebba44e82409852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 878235fb05a24dd48d48525ad7dbf3a5 in term 0.
I20260810 18:34:40.112944 26658 raft_consensus.cc:2781] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:34:40.113560 26658 raft_consensus.cc:482] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:34:40.113898 26658 raft_consensus.cc:3037] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:34:40.114794 26679 raft_consensus.cc:686] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5 [term 1 LEADER]: Becoming Leader. State: Replica: 878235fb05a24dd48d48525ad7dbf3a5, State: Running, Role: LEADER
I20260810 18:34:40.116106 26679 consensus_queue.cc:237] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
I20260810 18:34:40.118883 26658 raft_consensus.cc:504] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
I20260810 18:34:40.120234 26658 leader_election.cc:288] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5 [CANDIDATE]: Term 1 election: Requested vote from peers 31cbb92d24564804aebba44e82409852 (127.25.80.1:41319), 5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545)
I20260810 18:34:40.120298 26169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ffbeab79c14b74b8d98a6fad1fbfa9" candidate_uuid: "878235fb05a24dd48d48525ad7dbf3a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31cbb92d24564804aebba44e82409852"
I20260810 18:34:40.120639 26169 raft_consensus.cc:3037] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 31cbb92d24564804aebba44e82409852 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:34:40.119812 25968 catalog_manager.cc:5495] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f58fd4898774f4db897e97cc1598adf (127.25.80.3). New cstate: current_term: 1 leader_uuid: "5f58fd4898774f4db897e97cc1598adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } health_report { overall_health: UNKNOWN } } }
I20260810 18:34:40.121711 26566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ffbeab79c14b74b8d98a6fad1fbfa9" candidate_uuid: "878235fb05a24dd48d48525ad7dbf3a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f58fd4898774f4db897e97cc1598adf"
I20260810 18:34:40.122134 26566 raft_consensus.cc:3037] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:34:40.125797 26566 raft_consensus.cc:2454] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 878235fb05a24dd48d48525ad7dbf3a5 in term 1.
I20260810 18:34:40.126348 26256 leader_election.cc:302] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f58fd4898774f4db897e97cc1598adf, 878235fb05a24dd48d48525ad7dbf3a5; no voters:
I20260810 18:34:40.127053 26658 raft_consensus.cc:2781] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:34:40.127341 26658 raft_consensus.cc:686] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5 [term 1 LEADER]: Becoming Leader. State: Replica: 878235fb05a24dd48d48525ad7dbf3a5, State: Running, Role: LEADER
I20260810 18:34:40.127816 26658 consensus_queue.cc:237] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
I20260810 18:34:40.128295 26169 raft_consensus.cc:2454] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 31cbb92d24564804aebba44e82409852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 878235fb05a24dd48d48525ad7dbf3a5 in term 1.
I20260810 18:34:40.132398 25968 catalog_manager.cc:5495] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2). New cstate: current_term: 1 leader_uuid: "878235fb05a24dd48d48525ad7dbf3a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } health_report { overall_health: UNKNOWN } } }
I20260810 18:34:40.137614 25968 catalog_manager.cc:5495] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2). New cstate: current_term: 1 leader_uuid: "878235fb05a24dd48d48525ad7dbf3a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } health_report { overall_health: UNKNOWN } } }
W20260810 18:34:40.233716 26625 tablet.cc:2369] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:34:40.249533 26232 tablet.cc:2369] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 31cbb92d24564804aebba44e82409852: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:34:40.556703 26658 consensus_queue.cc:1035] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:34:40.567520 26679 consensus_queue.cc:1035] T 58bd2e2d08644a7883215728907d24b4 P 878235fb05a24dd48d48525ad7dbf3a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:34:40.574357 26670 consensus_queue.cc:1035] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:34:40.580678 26659 consensus_queue.cc:1035] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:34:40.642297 26679 consensus_queue.cc:1035] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:34:40.647239 26658 consensus_queue.cc:1035] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 878235fb05a24dd48d48525ad7dbf3a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:34:42.490026 26625 maintenance_manager.cc:419] P 5f58fd4898774f4db897e97cc1598adf: Scheduling FlushMRSOp(cfeba2dcf4954505b96a807ac76f55ce): perf score=1.000000
I20260810 18:34:42.672580 26711 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:49692 (ReqId={client: d7fa17e099144b1eb43b71176e090a6a, seq_no=1, attempt_no=0}) took 1101 ms. Trace:
I20260810 18:34:42.672995 26711 rpcz_store.cc:276] 0810 18:34:41.571192 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 18:34:41.571271 (+ 79us) service_pool.cc:227] Handling call
0810 18:34:42.672425 (+1101154us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 18:34:41.596042 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cfeba2dcf4954505b96a807ac76f55ce
0810 18:34:41.596535 (+ 493us) write_op.cc:432] Acquiring schema lock in shared mode
0810 18:34:41.596544 (+ 9us) write_op.cc:435] Acquired schema lock
0810 18:34:41.596546 (+ 2us) tablet.cc:660] Decoding operations
0810 18:34:41.882319 (+285773us) write_op.cc:620] Acquiring the partition lock for write op
0810 18:34:41.882359 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0810 18:34:41.882368 (+ 9us) tablet.cc:683] Acquiring locks for 10000 operations
0810 18:34:41.984623 (+102255us) tablet.cc:699] Row locks acquired
0810 18:34:41.984627 (+ 4us) write_op.cc:260] PREPARE: finished
0810 18:34:41.984754 (+ 127us) write_op.cc:270] Start()
0810 18:34:41.984810 (+ 56us) write_op.cc:276] Timestamp: P: 1786386881984730 usec, L: 0
0810 18:34:41.984875 (+ 65us) op_driver.cc:331] REPLICATION: starting
0810 18:34:41.992834 (+ 7959us) log.cc:842] Serialized 1527299 byte log entry
0810 18:34:42.074506 (+ 81672us) op_driver.cc:444] REPLICATION: finished
0810 18:34:42.074683 (+ 177us) write_op.cc:301] APPLY: starting
0810 18:34:42.074706 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0810 18:34:42.111054 (+ 36348us) tablet.cc:1368] finished BulkCheckPresence
0810 18:34:42.111148 (+ 94us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 18:34:42.585283 (+474135us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 18:34:42.591252 (+ 5969us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 18:34:42.591263 (+ 11us) write_op.cc:312] APPLY: finished
0810 18:34:42.626789 (+ 35526us) log.cc:842] Serialized 60022 byte log entry
0810 18:34:42.639803 (+ 13014us) write_op.cc:489] Releasing partition, row and schema locks
0810 18:34:42.671231 (+ 31428us) write_op.cc:454] Released schema lock
0810 18:34:42.671979 (+ 748us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":105,"num_ops":10000,"prepare.queue_time_us":12168,"prepare.run_cpu_time_us":323311,"prepare.run_wall_time_us":400846,"raft.queue_time_us":2372,"raft.run_cpu_time_us":22595,"raft.run_wall_time_us":40639,"replication_time_us":89595,"thread_start_us":16641,"threads_started":3,"wal-append.queue_time_us":1394}]]}
I20260810 18:34:42.746196 26437 maintenance_manager.cc:419] P 878235fb05a24dd48d48525ad7dbf3a5: Scheduling FlushMRSOp(cfeba2dcf4954505b96a807ac76f55ce): perf score=1.000000
I20260810 18:34:42.756152 26232 maintenance_manager.cc:419] P 31cbb92d24564804aebba44e82409852: Scheduling FlushMRSOp(cfeba2dcf4954505b96a807ac76f55ce): perf score=1.000000
W20260810 18:34:43.194832 26630 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.25.80.3:40545, user_credentials={real_user=slave}} blocked reactor thread for 518808us
W20260810 18:34:43.519822 26262 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-1/data/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-1/data/data (502312960 bytes available vs 518351011 bytes reserved) (error 28)
W20260810 18:34:43.605065 26262 tablet_replica_mm_ops.cc:214] T cfeba2dcf4954505b96a807ac76f55ce P 878235fb05a24dd48d48525ad7dbf3a5: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: tablet cfeba2dcf4954505b96a807ac76f55ce: unable to open output file for column bool_val BOOL NULLABLE: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in cfeba2dcf4954505b96a807ac76f55ce's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
F20260810 18:34:43.605533 26262 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: tablet cfeba2dcf4954505b96a807ac76f55ce: unable to open output file for column bool_val BOOL NULLABLE: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in cfeba2dcf4954505b96a807ac76f55ce's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
*** Check failure stack trace: ***
@ 0x7f627c4b69ad google::LogMessage::Fail() at ??:0
@ 0x7f627c4b988a google::LogMessage::SendToLog() at ??:0
@ 0x7f627c4b64de google::LogMessage::Flush() at ??:0
@ 0x7f627c4b7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f628b88be7d kudu::tablet::FlushMRSOp::Perform() at ??:0
@ 0x7f627efa4090 kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7f627efa6e53 kudu::MaintenanceManager::RunSchedulerThread()::$_3::operator()() at ??:0
@ 0x7f627efa6ce9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f627f102345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f627f10591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f627f105799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f627f0e0eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f62823ba6db start_thread at ??:0
@ 0x7f627b35061f clone at ??:0
W20260810 18:34:44.478173 26078 connection.cc:535] server connection from 127.25.80.2:51975 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 18:34:44.479099 26633 connection.cc:535] client connection to 127.25.80.2:44027 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 18:34:44.479553 26633 meta_cache.cc:302] tablet 58bd2e2d08644a7883215728907d24b4: replica 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027) has failed: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 18:34:44.480494 26473 connection.cc:535] client connection to 127.25.80.2:44027 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 18:34:44.480769 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 18:34:44.510111 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260810 18:34:44.526113 26115 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36726: Illegal state: replica 31cbb92d24564804aebba44e82409852 is not leader of this config: current role FOLLOWER
W20260810 18:34:44.539583 26115 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36726: Illegal state: replica 31cbb92d24564804aebba44e82409852 is not leader of this config: current role FOLLOWER
W20260810 18:34:44.587061 26632 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.25.80.1:41319, user_credentials={real_user=slave}} blocked reactor thread for 62301.7us
W20260810 18:34:44.599289 26507 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49692: Illegal state: replica 5f58fd4898774f4db897e97cc1598adf is not leader of this config: current role FOLLOWER
W20260810 18:34:44.602788 26506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49692: Illegal state: replica 5f58fd4898774f4db897e97cc1598adf is not leader of this config: current role FOLLOWER
W20260810 18:34:44.633998 26115 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36726: Illegal state: replica 31cbb92d24564804aebba44e82409852 is not leader of this config: current role FOLLOWER
W20260810 18:34:44.651613 26506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49692: Illegal state: replica 5f58fd4898774f4db897e97cc1598adf is not leader of this config: current role FOLLOWER
W20260810 18:34:44.654464 26115 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36726: Illegal state: replica 31cbb92d24564804aebba44e82409852 is not leader of this config: current role FOLLOWER
W20260810 18:34:44.671828 26506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49692: Illegal state: replica 5f58fd4898774f4db897e97cc1598adf is not leader of this config: current role FOLLOWER
W20260810 18:34:44.686896 26115 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36726: Illegal state: replica 31cbb92d24564804aebba44e82409852 is not leader of this config: current role FOLLOWER
W20260810 18:34:44.705150 26506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49692: Illegal state: replica 5f58fd4898774f4db897e97cc1598adf is not leader of this config: current role FOLLOWER
W20260810 18:34:44.715387 26115 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36726: Illegal state: replica 31cbb92d24564804aebba44e82409852 is not leader of this config: current role FOLLOWER
W20260810 18:34:44.739445 26115 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36726: Illegal state: replica 31cbb92d24564804aebba44e82409852 is not leader of this config: current role FOLLOWER
W20260810 18:34:44.751400 26506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49692: Illegal state: replica 5f58fd4898774f4db897e97cc1598adf is not leader of this config: current role FOLLOWER
W20260810 18:34:44.758599 26506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49692: Illegal state: replica 5f58fd4898774f4db897e97cc1598adf is not leader of this config: current role FOLLOWER
W20260810 18:34:44.796123 26115 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36726: Illegal state: replica 31cbb92d24564804aebba44e82409852 is not leader of this config: current role FOLLOWER
W20260810 18:34:44.807010 26115 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36726: Illegal state: replica 31cbb92d24564804aebba44e82409852 is not leader of this config: current role FOLLOWER
W20260810 18:34:44.819875 26506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49692: Illegal state: replica 5f58fd4898774f4db897e97cc1598adf is not leader of this config: current role FOLLOWER
W20260810 18:34:44.836299 26506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49692: Illegal state: replica 5f58fd4898774f4db897e97cc1598adf is not leader of this config: current role FOLLOWER
W20260810 18:34:44.868000 26115 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36726: Illegal state: replica 31cbb92d24564804aebba44e82409852 is not leader of this config: current role FOLLOWER
W20260810 18:34:44.917312 26506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49692: Illegal state: replica 5f58fd4898774f4db897e97cc1598adf is not leader of this config: current role FOLLOWER
W20260810 18:34:44.945101 26115 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36726: Illegal state: replica 31cbb92d24564804aebba44e82409852 is not leader of this config: current role FOLLOWER
I20260810 18:34:44.965207 26713 raft_consensus.cc:482] T 58bd2e2d08644a7883215728907d24b4 P 5f58fd4898774f4db897e97cc1598adf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 878235fb05a24dd48d48525ad7dbf3a5)
I20260810 18:34:44.966325 26713 raft_consensus.cc:504] T 58bd2e2d08644a7883215728907d24b4 P 5f58fd4898774f4db897e97cc1598adf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
I20260810 18:34:44.984370 26713 leader_election.cc:288] T 58bd2e2d08644a7883215728907d24b4 P 5f58fd4898774f4db897e97cc1598adf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31cbb92d24564804aebba44e82409852 (127.25.80.1:41319), 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027)
W20260810 18:34:44.986388 26473 leader_election.cc:334] T 58bd2e2d08644a7883215728907d24b4 P 5f58fd4898774f4db897e97cc1598adf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111)
I20260810 18:34:44.989948 26169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bd2e2d08644a7883215728907d24b4" candidate_uuid: "5f58fd4898774f4db897e97cc1598adf" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "31cbb92d24564804aebba44e82409852" is_pre_election: true
I20260810 18:34:44.990918 26472 leader_election.cc:302] T 58bd2e2d08644a7883215728907d24b4 P 5f58fd4898774f4db897e97cc1598adf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f58fd4898774f4db897e97cc1598adf; no voters: 31cbb92d24564804aebba44e82409852, 878235fb05a24dd48d48525ad7dbf3a5
I20260810 18:34:44.991575 26713 raft_consensus.cc:2726] T 58bd2e2d08644a7883215728907d24b4 P 5f58fd4898774f4db897e97cc1598adf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 18:34:44.993113 26115 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36726: Illegal state: replica 31cbb92d24564804aebba44e82409852 is not leader of this config: current role FOLLOWER
W20260810 18:34:45.014297 26506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49692: Illegal state: replica 5f58fd4898774f4db897e97cc1598adf is not leader of this config: current role FOLLOWER
W20260810 18:34:45.052937 26506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49692: Illegal state: replica 5f58fd4898774f4db897e97cc1598adf is not leader of this config: current role FOLLOWER
I20260810 18:34:45.112181 26773 raft_consensus.cc:482] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 31cbb92d24564804aebba44e82409852 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 878235fb05a24dd48d48525ad7dbf3a5)
I20260810 18:34:45.112895 26773 raft_consensus.cc:504] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 31cbb92d24564804aebba44e82409852 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
W20260810 18:34:45.116243 26115 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36726: Illegal state: replica 31cbb92d24564804aebba44e82409852 is not leader of this config: current role FOLLOWER
I20260810 18:34:45.133216 26773 leader_election.cc:288] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 31cbb92d24564804aebba44e82409852 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027), 5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545)
W20260810 18:34:45.187662 26506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49692: Illegal state: replica 5f58fd4898774f4db897e97cc1598adf is not leader of this config: current role FOLLOWER
W20260810 18:34:45.201390 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111)
W20260810 18:34:45.205014 26115 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36726: Illegal state: replica 31cbb92d24564804aebba44e82409852 is not leader of this config: current role FOLLOWER
I20260810 18:34:45.208300 26566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ffbeab79c14b74b8d98a6fad1fbfa9" candidate_uuid: "31cbb92d24564804aebba44e82409852" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5f58fd4898774f4db897e97cc1598adf" is_pre_election: true
W20260810 18:34:45.227820 26079 leader_election.cc:334] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 31cbb92d24564804aebba44e82409852 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111)
I20260810 18:34:45.228340 26079 leader_election.cc:302] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 31cbb92d24564804aebba44e82409852 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31cbb92d24564804aebba44e82409852; no voters: 5f58fd4898774f4db897e97cc1598adf, 878235fb05a24dd48d48525ad7dbf3a5
I20260810 18:34:45.228991 26773 raft_consensus.cc:2726] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 31cbb92d24564804aebba44e82409852 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 18:34:45.264793 26506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49692: Illegal state: replica 5f58fd4898774f4db897e97cc1598adf is not leader of this config: current role FOLLOWER
W20260810 18:34:45.271543 26115 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36726: Illegal state: replica 31cbb92d24564804aebba44e82409852 is not leader of this config: current role FOLLOWER
I20260810 18:34:45.326821 26713 raft_consensus.cc:482] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 878235fb05a24dd48d48525ad7dbf3a5)
I20260810 18:34:45.327143 26713 raft_consensus.cc:504] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
I20260810 18:34:45.328027 26713 leader_election.cc:288] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31cbb92d24564804aebba44e82409852 (127.25.80.1:41319), 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027)
I20260810 18:34:45.328989 26169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ffbeab79c14b74b8d98a6fad1fbfa9" candidate_uuid: "5f58fd4898774f4db897e97cc1598adf" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "31cbb92d24564804aebba44e82409852" is_pre_election: true
I20260810 18:34:45.329283 26169 raft_consensus.cc:2454] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 31cbb92d24564804aebba44e82409852 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f58fd4898774f4db897e97cc1598adf in term 1.
I20260810 18:34:45.329792 26472 leader_election.cc:302] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31cbb92d24564804aebba44e82409852, 5f58fd4898774f4db897e97cc1598adf; no voters:
I20260810 18:34:45.330267 26713 raft_consensus.cc:2781] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:34:45.330426 26713 raft_consensus.cc:482] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 878235fb05a24dd48d48525ad7dbf3a5)
I20260810 18:34:45.330564 26713 raft_consensus.cc:3037] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf [term 1 FOLLOWER]: Advancing to term 2
W20260810 18:34:45.333969 26473 leader_election.cc:334] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111)
I20260810 18:34:45.334321 26713 raft_consensus.cc:504] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
I20260810 18:34:45.335680 26169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ffbeab79c14b74b8d98a6fad1fbfa9" candidate_uuid: "5f58fd4898774f4db897e97cc1598adf" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "31cbb92d24564804aebba44e82409852"
I20260810 18:34:45.335981 26169 raft_consensus.cc:3037] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 31cbb92d24564804aebba44e82409852 [term 1 FOLLOWER]: Advancing to term 2
W20260810 18:34:45.337563 26473 leader_election.cc:334] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111)
I20260810 18:34:45.337886 26713 leader_election.cc:288] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf [CANDIDATE]: Term 2 election: Requested vote from peers 31cbb92d24564804aebba44e82409852 (127.25.80.1:41319), 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027)
I20260810 18:34:45.340059 26169 raft_consensus.cc:2454] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 31cbb92d24564804aebba44e82409852 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f58fd4898774f4db897e97cc1598adf in term 2.
I20260810 18:34:45.341382 26472 leader_election.cc:302] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31cbb92d24564804aebba44e82409852, 5f58fd4898774f4db897e97cc1598adf; no voters: 878235fb05a24dd48d48525ad7dbf3a5
W20260810 18:34:45.341686 26115 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36726: Illegal state: replica 31cbb92d24564804aebba44e82409852 is not leader of this config: current role FOLLOWER
I20260810 18:34:45.341967 26713 raft_consensus.cc:2781] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:34:45.342175 26713 raft_consensus.cc:686] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf [term 2 LEADER]: Becoming Leader. State: Replica: 5f58fd4898774f4db897e97cc1598adf, State: Running, Role: LEADER
I20260810 18:34:45.342631 26713 consensus_queue.cc:237] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
W20260810 18:34:45.345222 26506 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49692: Illegal state: replica 5f58fd4898774f4db897e97cc1598adf is not leader of this config: current role FOLLOWER
I20260810 18:34:45.356035 25970 catalog_manager.cc:5495] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf reported cstate change: term changed from 1 to 2, leader changed from 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2) to 5f58fd4898774f4db897e97cc1598adf (127.25.80.3). New cstate: current_term: 2 leader_uuid: "5f58fd4898774f4db897e97cc1598adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } health_report { overall_health: HEALTHY } } }
I20260810 18:34:45.360788 26773 raft_consensus.cc:482] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 878235fb05a24dd48d48525ad7dbf3a5)
I20260810 18:34:45.361027 26773 raft_consensus.cc:504] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
I20260810 18:34:45.361925 26773 leader_election.cc:288] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027), 5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545)
I20260810 18:34:45.362406 26566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bd2e2d08644a7883215728907d24b4" candidate_uuid: "31cbb92d24564804aebba44e82409852" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5f58fd4898774f4db897e97cc1598adf" is_pre_election: true
I20260810 18:34:45.362712 26566 raft_consensus.cc:2454] T 58bd2e2d08644a7883215728907d24b4 P 5f58fd4898774f4db897e97cc1598adf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31cbb92d24564804aebba44e82409852 in term 1.
I20260810 18:34:45.363196 26076 leader_election.cc:302] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31cbb92d24564804aebba44e82409852, 5f58fd4898774f4db897e97cc1598adf; no voters:
I20260810 18:34:45.363524 26773 raft_consensus.cc:2781] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:34:45.363680 26773 raft_consensus.cc:482] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 878235fb05a24dd48d48525ad7dbf3a5)
I20260810 18:34:45.363812 26773 raft_consensus.cc:3037] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:34:45.367601 26773 raft_consensus.cc:504] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
I20260810 18:34:45.368469 26773 leader_election.cc:288] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 [CANDIDATE]: Term 2 election: Requested vote from peers 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027), 5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545)
I20260810 18:34:45.369005 26566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bd2e2d08644a7883215728907d24b4" candidate_uuid: "31cbb92d24564804aebba44e82409852" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5f58fd4898774f4db897e97cc1598adf"
I20260810 18:34:45.369360 26566 raft_consensus.cc:3037] T 58bd2e2d08644a7883215728907d24b4 P 5f58fd4898774f4db897e97cc1598adf [term 1 FOLLOWER]: Advancing to term 2
W20260810 18:34:45.369824 26079 leader_election.cc:334] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111)
W20260810 18:34:45.371086 26079 leader_election.cc:334] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111)
I20260810 18:34:45.372758 26566 raft_consensus.cc:2454] T 58bd2e2d08644a7883215728907d24b4 P 5f58fd4898774f4db897e97cc1598adf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31cbb92d24564804aebba44e82409852 in term 2.
I20260810 18:34:45.373262 26076 leader_election.cc:302] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31cbb92d24564804aebba44e82409852, 5f58fd4898774f4db897e97cc1598adf; no voters: 878235fb05a24dd48d48525ad7dbf3a5
I20260810 18:34:45.373850 26773 raft_consensus.cc:2781] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:34:45.389489 26773 raft_consensus.cc:686] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 [term 2 LEADER]: Becoming Leader. State: Replica: 31cbb92d24564804aebba44e82409852, State: Running, Role: LEADER
I20260810 18:34:45.390435 26773 consensus_queue.cc:237] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } }
I20260810 18:34:45.408852 25970 catalog_manager.cc:5495] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 reported cstate change: term changed from 1 to 2, leader changed from 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2) to 31cbb92d24564804aebba44e82409852 (127.25.80.1). New cstate: current_term: 2 leader_uuid: "31cbb92d24564804aebba44e82409852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "878235fb05a24dd48d48525ad7dbf3a5" member_type: VOTER last_known_addr { host: "127.25.80.2" port: 44027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 } health_report { overall_health: UNKNOWN } } }
I20260810 18:34:45.605703 26169 raft_consensus.cc:1261] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 31cbb92d24564804aebba44e82409852 [term 2 FOLLOWER]: Refusing update from remote peer 5f58fd4898774f4db897e97cc1598adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20260810 18:34:45.606007 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 18:34:45.606705 26713 consensus_queue.cc:1035] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "31cbb92d24564804aebba44e82409852" member_type: VOTER last_known_addr { host: "127.25.80.1" port: 41319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:34:45.658538 26789 mvcc.cc:205] Tried to move back new op lower bound from 7317040683416891392 to 7317040682367537152. Current Snapshot: MvccSnapshot[applied={T|T < 7317040683416891392}]
I20260810 18:34:45.686223 26566 raft_consensus.cc:1261] T 58bd2e2d08644a7883215728907d24b4 P 5f58fd4898774f4db897e97cc1598adf [term 2 FOLLOWER]: Refusing update from remote peer 31cbb92d24564804aebba44e82409852: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260810 18:34:45.688776 26784 consensus_queue.cc:1035] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f58fd4898774f4db897e97cc1598adf" member_type: VOTER last_known_addr { host: "127.25.80.3" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260810 18:34:45.703037 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 18:34:46.025535 26794 mvcc.cc:205] Tried to move back new op lower bound from 7317040683416891392 to 7317040682367537152. Current Snapshot: MvccSnapshot[applied={T|T < 7317040683416891392}]
W20260810 18:34:46.147917 26630 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.25.80.3:40545, user_credentials={real_user=slave}} blocked reactor thread for 70668us
I20260810 18:34:46.360188 26474 maintenance_manager.cc:643] P 5f58fd4898774f4db897e97cc1598adf: FlushMRSOp(cfeba2dcf4954505b96a807ac76f55ce) complete. Timing: real 3.869s user 1.414s sys 0.476s Metrics: {"bytes_written":436310,"cfile_init":1,"compiler_manager_pool.queue_time_us":20128,"compiler_manager_pool.run_cpu_time_us":1315625,"compiler_manager_pool.run_wall_time_us":2785910,"dirs.queue_time_us":7100,"dirs.run_cpu_time_us":238,"dirs.run_wall_time_us":2597,"drs_written":1,"lbm_read_time_us":141,"lbm_reads_lt_1ms":4,"lbm_write_time_us":42901,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":84,"peak_mem_usage":0,"rows_written":10000,"spinlock_wait_cycles":254720,"thread_start_us":44655,"threads_started":3,"wal-append.queue_time_us":8992}
I20260810 18:34:46.362540 26625 maintenance_manager.cc:419] P 5f58fd4898774f4db897e97cc1598adf: Scheduling UndoDeltaBlockGCOp(cfeba2dcf4954505b96a807ac76f55ce): 62923 bytes on disk
I20260810 18:34:46.370415 26474 maintenance_manager.cc:643] P 5f58fd4898774f4db897e97cc1598adf: UndoDeltaBlockGCOp(cfeba2dcf4954505b96a807ac76f55ce) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20260810 18:34:46.371652 26625 maintenance_manager.cc:419] P 5f58fd4898774f4db897e97cc1598adf: Scheduling FlushMRSOp(35ffbeab79c14b74b8d98a6fad1fbfa9): perf score=1.937278
W20260810 18:34:46.398357 26632 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.25.80.1:41319, user_credentials={real_user=slave}} blocked reactor thread for 108532us
I20260810 18:34:46.421622 26080 maintenance_manager.cc:643] P 31cbb92d24564804aebba44e82409852: FlushMRSOp(cfeba2dcf4954505b96a807ac76f55ce) complete. Timing: real 3.664s user 1.359s sys 0.522s Metrics: {"bytes_written":436310,"cfile_init":1,"compiler_manager_pool.queue_time_us":17817,"compiler_manager_pool.run_cpu_time_us":1387583,"compiler_manager_pool.run_wall_time_us":2569998,"dirs.queue_time_us":37174,"dirs.run_cpu_time_us":292,"dirs.run_wall_time_us":953,"drs_written":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7619,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":85,"peak_mem_usage":0,"rows_written":10000,"spinlock_wait_cycles":305408,"thread_start_us":54939,"threads_started":3,"wal-append.queue_time_us":410}
I20260810 18:34:46.423504 26232 maintenance_manager.cc:419] P 31cbb92d24564804aebba44e82409852: Scheduling FlushMRSOp(35ffbeab79c14b74b8d98a6fad1fbfa9): perf score=1.937278
I20260810 18:34:46.540122 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:34:46.583211 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:34:46.652694 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 18:34:47.006958 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:34:47.020133 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:34:47.094630 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:34:47.173892 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260810 18:34:47.530143 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:34:47.544872 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:34:47.616217 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 18:34:47.616094 26080 maintenance_manager.cc:643] P 31cbb92d24564804aebba44e82409852: FlushMRSOp(35ffbeab79c14b74b8d98a6fad1fbfa9) complete. Timing: real 1.192s user 0.844s sys 0.320s Metrics: {"bytes_written":471972,"cfile_init":1,"dirs.queue_time_us":544,"dirs.run_cpu_time_us":306,"dirs.run_wall_time_us":1310,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5083,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":896,"threads_started":2,"wal-append.queue_time_us":572}
I20260810 18:34:47.616851 26232 maintenance_manager.cc:419] P 31cbb92d24564804aebba44e82409852: Scheduling UndoDeltaBlockGCOp(35ffbeab79c14b74b8d98a6fad1fbfa9): 63424 bytes on disk
I20260810 18:34:47.617866 26080 maintenance_manager.cc:643] P 31cbb92d24564804aebba44e82409852: UndoDeltaBlockGCOp(35ffbeab79c14b74b8d98a6fad1fbfa9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20260810 18:34:47.618443 26232 maintenance_manager.cc:419] P 31cbb92d24564804aebba44e82409852: Scheduling UndoDeltaBlockGCOp(cfeba2dcf4954505b96a807ac76f55ce): 62923 bytes on disk
I20260810 18:34:47.618885 26080 maintenance_manager.cc:643] P 31cbb92d24564804aebba44e82409852: UndoDeltaBlockGCOp(cfeba2dcf4954505b96a807ac76f55ce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20260810 18:34:47.619372 26232 maintenance_manager.cc:419] P 31cbb92d24564804aebba44e82409852: Scheduling FlushMRSOp(58bd2e2d08644a7883215728907d24b4): perf score=1.937278
I20260810 18:34:47.630733 26474 maintenance_manager.cc:643] P 5f58fd4898774f4db897e97cc1598adf: FlushMRSOp(35ffbeab79c14b74b8d98a6fad1fbfa9) complete. Timing: real 1.259s user 0.822s sys 0.384s Metrics: {"bytes_written":471972,"cfile_init":1,"dirs.queue_time_us":408,"dirs.run_cpu_time_us":266,"dirs.run_wall_time_us":1374,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5386,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":611,"threads_started":2,"wal-append.queue_time_us":498}
I20260810 18:34:47.631779 26625 maintenance_manager.cc:419] P 5f58fd4898774f4db897e97cc1598adf: Scheduling FlushMRSOp(58bd2e2d08644a7883215728907d24b4): perf score=1.937278
W20260810 18:34:48.128522 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260810 18:34:48.146299 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:34:48.155692 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:34:48.210031 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 18:34:48.426777 26080 maintenance_manager.cc:643] P 31cbb92d24564804aebba44e82409852: FlushMRSOp(58bd2e2d08644a7883215728907d24b4) complete. Timing: real 0.807s user 0.667s sys 0.124s Metrics: {"bytes_written":472920,"cfile_init":1,"dirs.queue_time_us":437,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":1551,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5278,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":673,"threads_started":2,"wal-append.queue_time_us":483}
I20260810 18:34:48.427608 26232 maintenance_manager.cc:419] P 31cbb92d24564804aebba44e82409852: Scheduling UndoDeltaBlockGCOp(58bd2e2d08644a7883215728907d24b4): 63422 bytes on disk
I20260810 18:34:48.428179 26080 maintenance_manager.cc:643] P 31cbb92d24564804aebba44e82409852: UndoDeltaBlockGCOp(58bd2e2d08644a7883215728907d24b4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
W20260810 18:34:48.472625 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260810 18:34:48.492834 26474 maintenance_manager.cc:643] P 5f58fd4898774f4db897e97cc1598adf: FlushMRSOp(58bd2e2d08644a7883215728907d24b4) complete. Timing: real 0.861s user 0.713s sys 0.139s Metrics: {"bytes_written":472920,"cfile_init":1,"dirs.queue_time_us":420,"dirs.run_cpu_time_us":262,"dirs.run_wall_time_us":1652,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4158,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":517,"threads_started":2,"wal-append.queue_time_us":316}
I20260810 18:34:48.493793 26625 maintenance_manager.cc:419] P 5f58fd4898774f4db897e97cc1598adf: Scheduling UndoDeltaBlockGCOp(35ffbeab79c14b74b8d98a6fad1fbfa9): 63424 bytes on disk
I20260810 18:34:48.494338 26474 maintenance_manager.cc:643] P 5f58fd4898774f4db897e97cc1598adf: UndoDeltaBlockGCOp(35ffbeab79c14b74b8d98a6fad1fbfa9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20260810 18:34:48.494887 26625 maintenance_manager.cc:419] P 5f58fd4898774f4db897e97cc1598adf: Scheduling UndoDeltaBlockGCOp(58bd2e2d08644a7883215728907d24b4): 63422 bytes on disk
I20260810 18:34:48.495309 26474 maintenance_manager.cc:643] P 5f58fd4898774f4db897e97cc1598adf: UndoDeltaBlockGCOp(58bd2e2d08644a7883215728907d24b4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20260810 18:34:48.808678 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:34:48.810715 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:34:48.843446 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:34:49.621436 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20260810 18:34:49.687597 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:34:49.688403 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
W20260810 18:34:49.722101 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:34:49.730772 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:34:50.379460 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20260810 18:34:50.472160 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260810 18:34:50.818559 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260810 18:34:50.826558 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:34:50.827694 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:34:50.859673 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 18:34:51.953997 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:34:51.954422 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:34:51.998735 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:34:52.284610 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260810 18:34:53.072367 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260810 18:34:53.099015 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:34:53.102588 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20260810 18:34:53.129330 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:34:53.148052 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 18:34:54.266849 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:34:54.267131 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:34:54.314164 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:34:54.886296 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:34:54.887277 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260810 18:34:55.400219 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:34:55.400278 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:34:55.433130 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:34:55.530597 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260810 18:34:55.779253 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:34:55.780520 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260810 18:34:56.521914 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:34:56.522044 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:34:56.554971 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:34:56.958257 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260810 18:34:57.642078 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:34:57.642894 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:34:57.687855 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:34:57.998310 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260810 18:34:58.247377 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260810 18:34:58.778450 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:34:58.783790 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:34:58.815632 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:34:59.332250 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260810 18:34:59.925406 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:34:59.937069 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:34:59.988713 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:00.000344 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 18:35:00.501549 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260810 18:35:00.527369 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260810 18:35:00.911989 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20260810 18:35:01.118549 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:01.122046 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:01.170809 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:01.553525 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260810 18:35:02.265269 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:02.265316 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:02.304064 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:02.864189 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260810 18:35:02.950131 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260810 18:35:03.388768 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:03.390488 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:03.431969 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:04.312279 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260810 18:35:04.527818 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:35:04.527834 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:04.571839 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:05.367412 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:35:05.368347 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260810 18:35:05.446926 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260810 18:35:05.659344 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:35:05.665757 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:05.696532 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:06.022787 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:35:06.672659 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260810 18:35:06.797487 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:35:06.798048 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:06.836686 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:07.894871 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260810 18:35:07.962564 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:07.974229 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:08.024626 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:08.138417 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260810 18:35:08.939946 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260810 18:35:09.138330 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:35:09.138372 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:09.174700 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 18:35:10.271381 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:10.273520 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:10.306264 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:10.310369 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260810 18:35:10.576992 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20260810 18:35:10.939011 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260810 18:35:11.065004 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260810 18:35:11.327621 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260810 18:35:11.395777 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:35:11.396416 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:11.427691 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 18:35:12.518767 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:35:12.519047 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:12.556454 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:12.601028 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260810 18:35:13.241070 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260810 18:35:13.405068 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260810 18:35:13.678529 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:35:13.679762 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:13.728425 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 18:35:14.842849 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:14.853690 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:14.895473 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:15.226389 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260810 18:35:15.523653 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260810 18:35:15.759847 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 18:35:15.924510 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260810 18:35:15.982700 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:35:15.982786 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:16.016644 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:16.457948 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260810 18:35:17.107771 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:17.108525 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:17.139459 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:17.967326 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260810 18:35:18.034704 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260810 18:35:18.246835 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:18.256402 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:18.309424 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:18.740448 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260810 18:35:19.442408 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:19.444283 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:19.508136 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:20.473126 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260810 18:35:20.631599 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:20.631652 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:20.689850 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:20.769773 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260810 18:35:20.770900 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260810 18:35:21.511059 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20260810 18:35:21.512537 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260810 18:35:21.785390 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:21.786092 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:21.818830 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 18:35:22.914693 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:35:22.914997 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:22.953042 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:23.031208 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20260810 18:35:23.056147 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260810 18:35:23.803876 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20260810 18:35:24.034464 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:24.035038 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:24.074136 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 18:35:25.161548 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:25.161988 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:25.194523 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:25.500484 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20260810 18:35:25.652206 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20260810 18:35:26.055851 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:35:26.140096 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20260810 18:35:26.345170 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:35:26.345700 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:26.405995 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:26.711665 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20260810 18:35:27.506943 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:27.523948 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:27.556248 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:27.942191 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20260810 18:35:28.250900 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20260810 18:35:28.667111 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:28.680755 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:28.727767 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:28.875823 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20260810 18:35:29.844413 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:29.847402 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:29.902848 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:30.221866 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20260810 18:35:30.676183 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20260810 18:35:30.988651 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:30.989854 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:31.024689 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:31.228103 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20260810 18:35:31.338505 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20260810 18:35:32.109823 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:35:32.109825 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:32.142855 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:32.242816 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:35:32.671118 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20260810 18:35:32.782023 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20260810 18:35:33.230633 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:33.230762 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:33.265544 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:33.753890 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20260810 18:35:34.389758 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:34.389828 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:34.430812 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:35.031378 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20260810 18:35:35.087867 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20260810 18:35:35.545328 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:35.546324 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:35.587782 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:36.293715 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20260810 18:35:36.415548 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20260810 18:35:36.701392 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:35:36.702239 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:36.745299 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:37.323945 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:35:37.413228 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20260810 18:35:37.547991 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20260810 18:35:37.848133 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:37.849817 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:37.886030 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 18:35:38.975553 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:35:38.975589 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20260810 18:35:38.998504 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:39.013847 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:39.740734 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20260810 18:35:40.106282 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:40.116405 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:40.155277 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:40.239588 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20260810 18:35:41.245702 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:41.252578 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:41.284258 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:41.332994 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20260810 18:35:41.593608 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:35:42.340912 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20260810 18:35:42.387681 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:42.389532 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20260810 18:35:42.422547 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:42.436520 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:42.470294 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20260810 18:35:43.550227 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:43.551460 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:43.601555 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:43.883824 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20260810 18:35:44.688642 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:35:44.688686 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:44.722528 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:44.901708 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20260810 18:35:44.994130 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20260810 18:35:45.816504 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:35:45.822132 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:45.868161 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:46.496048 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20260810 18:35:46.842183 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20260810 18:35:46.988085 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:35:46.994057 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:47.037014 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:47.295470 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20260810 18:35:47.590667 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20260810 18:35:47.788645 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20260810 18:35:48.157019 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:48.158665 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:48.214594 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:48.636343 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20260810 18:35:49.335146 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:49.336395 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:49.398674 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:49.715126 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20260810 18:35:50.255743 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20260810 18:35:50.515009 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:35:50.518142 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:50.561932 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:51.266520 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20260810 18:35:51.661032 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:35:51.661580 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:51.709100 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:51.844476 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:35:52.350170 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20260810 18:35:52.808063 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:52.810894 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:52.842631 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:53.006961 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20260810 18:35:53.122933 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:35:53.657977 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20260810 18:35:53.930893 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:53.943038 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:53.976205 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:54.693454 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20260810 18:35:55.063036 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:55.064548 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:55.103893 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:55.606838 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20260810 18:35:56.191828 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:56.195024 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:56.228499 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:56.311595 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20260810 18:35:57.074049 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:35:57.216746 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20260810 18:35:57.360809 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:57.361860 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:57.408200 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:58.206453 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20260810 18:35:58.500777 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:58.501251 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
W20260810 18:35:58.533644 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260810 18:35:58.536237 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:35:58.986379 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20260810 18:35:59.496055 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20260810 18:35:59.621629 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:35:59.622380 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:35:59.655572 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:00.463804 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20260810 18:36:00.754649 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:36:00.754755 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:00.792830 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:01.507735 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20260810 18:36:01.877606 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:36:01.877811 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:01.911566 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:02.301106 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:36:02.332616 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20260810 18:36:02.845829 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20260810 18:36:03.001078 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:36:03.002075 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:03.034090 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:03.731586 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:36:03.732753 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20260810 18:36:04.119320 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:36:04.121289 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:04.154575 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:04.744655 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20260810 18:36:05.240139 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:36:05.240535 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:05.277073 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:05.356344 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20260810 18:36:06.398602 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:36:06.407864 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:06.448452 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:06.488445 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20260810 18:36:07.407430 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:36:07.413550 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20260810 18:36:07.540570 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:36:07.542780 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:07.573854 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:08.018993 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20260810 18:36:08.689169 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:36:08.690677 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:08.731191 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:08.845781 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:36:08.846987 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20260810 18:36:09.844107 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20260810 18:36:09.855829 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:36:09.858199 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:09.897130 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:10.510155 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20260810 18:36:11.018905 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:36:11.022574 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:11.077208 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:11.565932 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20260810 18:36:12.177157 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:36:12.181754 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:12.215101 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:12.251659 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20260810 18:36:12.540486 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:36:13.023408 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20260810 18:36:13.325062 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:36:13.336694 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:13.385114 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:13.995069 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:36:13.996428 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20260810 18:36:14.471194 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:36:14.471238 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:14.505141 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:14.626389 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20260810 18:36:15.594978 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20260810 18:36:15.608018 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:36:15.620298 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:15.656879 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:16.514194 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20260810 18:36:16.756325 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:36:16.757926 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:16.798604 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:17.248660 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20260810 18:36:17.580664 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20260810 18:36:17.895565 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:36:17.895601 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:17.951491 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:18.169411 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20260810 18:36:19.055706 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:36:19.058133 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:19.098731 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:19.225317 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:36:19.226717 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20260810 18:36:19.445763 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20260810 18:36:20.236785 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:36:20.238541 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:20.285730 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:20.629712 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20260810 18:36:21.377079 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:36:21.377262 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:21.411139 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:21.461855 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20260810 18:36:21.934484 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20260810 18:36:22.507959 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:36:22.518565 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:22.562704 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:23.013942 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 18:36:23.014936 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20260810 18:36:23.650861 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:36:23.653223 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:23.700292 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:24.157793 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20260810 18:36:24.517107 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20260810 18:36:24.639652 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20260810 18:36:24.799510 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:36:24.805799 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:24.838606 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:25.731659 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20260810 18:36:25.928421 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:36:25.928987 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.3:40545, tserver@127.25.80.1:41319
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:25.962721 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:26.764274 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20260810 18:36:26.980225 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20260810 18:36:27.056432 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:36:27.067778 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:27.120519 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:28.197613 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:36:28.198619 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20260810 18:36:28.275467 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:36:28.277266 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:28.319281 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:29.230731 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20260810 18:36:29.415872 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:36:29.417197 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:29.455152 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:29.721968 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20260810 18:36:29.814889 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260810 18:36:30.551867 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:36:30.551904 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:30.590728 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:30.633836 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20260810 18:36:31.680446 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:36:31.686359 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:31.719803 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:31.765604 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20260810 18:36:32.524135 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20260810 18:36:32.825220 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 18:36:32.825278 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:32.857748 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:33.363427 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260810 18:36:33.364527 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20260810 18:36:33.963495 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:36:33.970583 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:34.014894 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:34.166911 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20260810 18:36:35.027518 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20260810 18:36:35.113842 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:36:35.122925 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:35.155398 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 18:36:35.225230 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:36:35.810359 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20260810 18:36:36.266361 26143 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:36:36.270522 26537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7fa3c861edac43bfb72fad37220c898f | 127.25.80.62:42753 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.25.80.20:40237 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
31cbb92d24564804aebba44e82409852 | 127.25.80.1:41319 | HEALTHY | <none> | 1 | 0
5f58fd4898774f4db897e97cc1598adf | 127.25.80.3:40545 | HEALTHY | <none> | 2 | 0
878235fb05a24dd48d48525ad7dbf3a5 | 127.25.80.2:44027 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.25.80.2:44027: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.25.80.1:41319, 127.25.80.3:40545
ipki_server_key_size | 768 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
local_ip_for_outbound_sockets | 127.25.80.1 | experimental | 127.25.80.1:41319
local_ip_for_outbound_sockets | 127.25.80.3 | experimental | 127.25.80.3:40545
never_fsync | true | unsafe,advanced | 127.25.80.1:41319, 127.25.80.3:40545
openssl_security_level_override | 0 | unsafe,hidden | 127.25.80.1:41319, 127.25.80.3:40545
rpc_server_allow_ephemeral_ports | true | unsafe | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_format | pb | hidden | 127.25.80.1:41319, 127.25.80.3:40545
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.80.1:41319
server_dump_info_path | /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.80.3:40545
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.25.80.1:41319, 127.25.80.3:40545
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.25.80.20:40237 | 127.25.80.1:41319, 127.25.80.3:40545
time_source | builtin | 127.25.80.1:41319, 127.25.80.3:40545
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.25.80.62:42753, tserver@127.25.80.1:41319, tserver@127.25.80.3:40545
Tablet Summary
Tablet cfeba2dcf4954505b96a807ac76f55ce of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 5f58fd4898774f4db897e97cc1598adf
C = 878235fb05a24dd48d48525ad7dbf3a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 35ffbeab79c14b74b8d98a6fad1fbfa9 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING [LEADER]
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 58bd2e2d08644a7883215728907d24b4 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
31cbb92d24564804aebba44e82409852 (127.25.80.1:41319): RUNNING [LEADER]
878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): TS unavailable
5f58fd4898774f4db897e97cc1598adf (127.25.80.3:40545): RUNNING
All reported replicas are:
A = 31cbb92d24564804aebba44e82409852
B = 878235fb05a24dd48d48525ad7dbf3a5
C = 5f58fd4898774f4db897e97cc1598adf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:36:36.307212 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 18:36:36.681814 25956 maintenance_manager.cc:419] P 7fa3c861edac43bfb72fad37220c898f: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033356
W20260810 18:36:36.685752 25955 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskJBH3sd/test-tmp/all_types-itest.2.AllTypesItest_1.TestAllKeyTypes.1786386875526509-25920-0/minicluster-data/master-0/data/data (497094656 bytes available vs 518351011 bytes reserved) (error 28)
W20260810 18:36:36.687011 25955 tablet_replica_mm_ops.cc:214] T 00000000000000000000000000000000 P 7fa3c861edac43bfb72fad37220c898f: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: tablet 00000000000000000000000000000000: unable to open output file for column entry_type INT8 NOT NULL: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 00000000000000000000000000000000's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
F20260810 18:36:36.687304 25955 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: tablet 00000000000000000000000000000000: unable to open output file for column entry_type INT8 NOT NULL: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 00000000000000000000000000000000's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
*** Check failure stack trace: ***
@ 0x7fb02b43a9ad google::LogMessage::Fail() at ??:0
@ 0x7fb02b43d88a google::LogMessage::SendToLog() at ??:0
@ 0x7fb02b43a4de google::LogMessage::Flush() at ??:0
@ 0x7fb02b43ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb03a80fe7d kudu::tablet::FlushMRSOp::Perform() at ??:0
@ 0x7fb02df28090 kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7fb02df2ae53 kudu::MaintenanceManager::RunSchedulerThread()::$_3::operator()() at ??:0
@ 0x7fb02df2ace9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb02e086345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb02e08991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb02e089799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb02e064eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb03133e6db start_thread at ??:0
W20260810 18:36:36.722682 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
@ 0x7fb02a2d461f clone at ??:0
W20260810 18:36:37.052459 26623 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:36:37.054697 26231 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:36:37.951336 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20260810 18:36:38.187276 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20260810 18:36:38.448761 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:36:39.212009 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20260810 18:36:40.258090 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:36:40.538800 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260810 18:36:40.824610 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20260810 18:36:41.741218 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20260810 18:36:42.908142 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260810 18:36:43.208992 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260810 18:36:43.660898 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:36:44.452446 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260810 18:36:45.285144 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:36:45.630020 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260810 18:36:45.669849 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260810 18:36:46.782359 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260810 18:36:48.118793 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260810 18:36:48.341203 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260810 18:36:48.795799 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:36:49.539171 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260810 18:36:50.483979 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:36:50.557204 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260810 18:36:50.890985 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260810 18:36:51.985275 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260810 18:36:53.051891 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260810 18:36:53.361878 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260810 18:36:53.826646 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:36:54.991029 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260810 18:36:55.538152 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20260810 18:36:55.631323 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260810 18:36:55.959311 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260810 18:36:57.432041 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260810 18:36:58.100102 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260810 18:36:58.824105 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260810 18:36:59.235615 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:36:59.825953 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260810 18:37:00.760473 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:37:00.800551 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260810 18:37:01.112910 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260810 18:37:02.517377 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260810 18:37:03.195972 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260810 18:37:03.496506 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260810 18:37:04.565433 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 18:37:05.352906 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260810 18:37:05.563110 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260810 18:37:05.853686 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260810 18:37:05.972589 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20260810 18:37:07.971402 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260810 18:37:07.997656 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260810 18:37:08.146026 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260810 18:37:09.592540 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 18:37:10.465171 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260810 18:37:10.596235 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260810 18:37:10.719610 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260810 18:37:11.026050 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:37:12.785277 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260810 18:37:13.161320 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260810 18:37:13.229188 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260810 18:37:14.720098 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:37:15.321847 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260810 18:37:15.755019 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260810 18:37:16.103874 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20260810 18:37:16.105180 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260810 18:37:17.637730 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260810 18:37:18.235277 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260810 18:37:19.026605 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260810 18:37:20.083114 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 18:37:20.084151 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260810 18:37:20.706559 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260810 18:37:21.347862 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:37:21.349092 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260810 18:37:22.424623 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260810 18:37:23.370817 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260810 18:37:23.904660 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260810 18:37:25.355767 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:37:25.356971 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260810 18:37:26.230134 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260810 18:37:26.527878 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:37:26.528898 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260810 18:37:27.936085 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260810 18:37:28.834939 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260810 18:37:29.074239 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260810 18:37:30.380468 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260810 18:37:30.617622 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260810 18:37:31.399389 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260810 18:37:31.524261 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260810 18:37:32.013980 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:37:33.046685 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260810 18:37:33.886277 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260810 18:37:34.046154 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260810 18:37:35.602643 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:37:35.603765 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260810 18:37:36.263706 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260810 18:37:36.289481 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260810 18:37:37.138222 26231 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (60 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:37:37.138252 26623 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (60 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:37:37.263432 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.25.80.62:42753) | 127.25.80.62:42753 | UNAVAILABLE
Error from 127.25.80.62:42753: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.25.80.62:42753, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
FAILED
I20260810 18:37:37.610648 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.25.80.62:42753, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:37:37.951512 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260810 18:37:38.630173 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260810 18:37:38.739172 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260810 18:37:40.269924 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260810 18:37:40.732638 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:37:41.132354 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260810 18:37:41.341602 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260810 18:37:42.486447 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:37:42.792377 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260810 18:37:43.307868 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260810 18:37:43.871599 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260810 18:37:45.620441 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260810 18:37:45.772221 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260810 18:37:46.029114 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:37:46.501582 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260810 18:37:47.863746 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:37:47.927459 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260810 18:37:48.368507 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260810 18:37:49.148625 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260810 18:37:50.491726 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260810 18:37:50.911626 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260810 18:37:51.426687 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 18:37:51.473699 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260810 18:37:52.894109 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260810 18:37:52.918520 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:37:53.310010 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260810 18:37:54.053174 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260810 18:37:55.469131 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260810 18:37:55.849121 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260810 18:37:56.528136 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:37:56.529381 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260810 18:37:57.970561 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:37:58.036819 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260810 18:37:58.499920 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260810 18:37:59.059695 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260810 18:38:00.604035 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260810 18:38:01.107353 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260810 18:38:01.322453 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260810 18:38:01.925658 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 18:38:03.106321 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260810 18:38:03.274408 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:38:03.870674 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260810 18:38:03.890627 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260810 18:38:05.146166 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260810 18:38:06.366304 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260810 18:38:06.541800 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260810 18:38:06.931438 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:38:07.665660 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260810 18:38:08.720093 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:38:09.052913 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260810 18:38:09.343463 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260810 18:38:10.034421 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260810 18:38:11.272089 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260810 18:38:11.752362 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260810 18:38:12.218703 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:38:12.689188 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260810 18:38:13.747979 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260810 18:38:14.072791 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:38:14.073872 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260810 18:38:15.025224 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260810 18:38:16.515261 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260810 18:38:16.542766 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260810 18:38:17.271780 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:38:17.883590 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260810 18:38:18.539935 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260810 18:38:19.194027 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:38:19.195184 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260810 18:38:20.291869 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260810 18:38:21.326171 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260810 18:38:21.537441 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260810 18:38:22.464069 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:38:22.842906 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260810 18:38:23.842010 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260810 18:38:23.863550 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260810 18:38:24.456499 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:38:25.106264 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260810 18:38:26.349159 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260810 18:38:26.664517 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260810 18:38:27.639765 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:38:27.640802 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260810 18:38:28.719581 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260810 18:38:28.967206 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260810 18:38:29.825814 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:38:30.102499 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260810 18:38:31.171685 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260810 18:38:31.398720 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260810 18:38:32.461515 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260810 18:38:32.923929 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 18:38:33.517257 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260810 18:38:34.044718 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260810 18:38:34.722460 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260810 18:38:35.332646 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:38:36.195966 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260810 18:38:36.463660 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260810 18:38:37.225618 26623 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (120 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:38:37.225970 26231 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (120 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:38:37.344652 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260810 18:38:38.157383 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:38:38.746506 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.25.80.62:42753) | 127.25.80.62:42753 | UNAVAILABLE
Error from 127.25.80.62:42753: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.25.80.62:42753, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
FAILED
I20260810 18:38:38.929147 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.25.80.62:42753, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:38:38.974296 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260810 18:38:39.774104 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260810 18:38:40.556823 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:38:40.942061 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260810 18:38:41.543551 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260810 18:38:42.048280 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260810 18:38:43.183981 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:38:43.555408 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260810 18:38:43.995157 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260810 18:38:44.474617 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260810 18:38:45.852466 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:38:45.853876 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260810 18:38:46.650877 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260810 18:38:47.067507 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260810 18:38:48.231752 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260810 18:38:48.504413 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:38:48.980019 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260810 18:38:49.507922 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260810 18:38:50.692185 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260810 18:38:51.240788 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:38:51.551435 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260810 18:38:52.002140 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260810 18:38:53.162348 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260810 18:38:53.519887 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:38:54.117689 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260810 18:38:54.704150 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260810 18:38:55.705475 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260810 18:38:56.604707 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:38:56.717815 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260810 18:38:57.221652 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260810 18:38:58.030109 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260810 18:38:58.670889 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:38:59.283373 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260810 18:38:59.688108 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260810 18:39:00.641993 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260810 18:39:01.746747 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:39:02.030615 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260810 18:39:02.460323 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260810 18:39:03.271579 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260810 18:39:03.851799 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:39:04.518903 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260810 18:39:04.917022 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260810 18:39:05.957791 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260810 18:39:06.808138 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:39:07.094086 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260810 18:39:07.469921 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260810 18:39:08.223664 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260810 18:39:08.952757 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:39:09.459506 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260810 18:39:10.180738 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260810 18:39:10.525887 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260810 18:39:11.811764 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:39:12.052140 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260810 18:39:12.661929 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260810 18:39:12.803844 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260810 18:39:13.953872 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:39:14.400703 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260810 18:39:15.122615 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260810 18:39:15.323117 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260810 18:39:16.920617 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260810 18:39:17.008436 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:39:17.725173 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260810 18:39:18.036119 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260810 18:39:19.072129 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:39:19.543601 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260810 18:39:20.170848 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260810 18:39:20.776095 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260810 18:39:22.010620 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260810 18:39:22.301851 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:39:22.654619 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260810 18:39:23.463858 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260810 18:39:24.240855 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:39:24.582393 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260810 18:39:25.229411 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260810 18:39:26.018993 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260810 18:39:27.122641 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260810 18:39:27.368432 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260810 18:39:27.581885 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:39:28.533495 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260810 18:39:29.576157 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:39:29.577247 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260810 18:39:30.249145 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260810 18:39:30.941859 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260810 18:39:31.819824 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260810 18:39:32.642527 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:39:32.866470 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260810 18:39:33.070175 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260810 18:39:34.267359 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260810 18:39:34.685442 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 18:39:35.080090 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260810 18:39:35.728816 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260810 18:39:36.645311 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260810 18:39:37.296026 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260810 18:39:37.315872 26231 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (180 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:39:37.316054 26623 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (180 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:39:37.771557 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:39:38.220196 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260810 18:39:39.286504 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260810 18:39:39.705705 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:39:39.721549 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.25.80.62:42753) | 127.25.80.62:42753 | UNAVAILABLE
Error from 127.25.80.62:42753: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.25.80.62:42753, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
FAILED
I20260810 18:39:40.211732 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.25.80.62:42753, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:39:40.599140 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260810 18:39:41.700838 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20260810 18:39:42.042424 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20260810 18:39:43.226392 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:39:43.227636 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20260810 18:39:43.720690 31335 consensus_queue.cc:579] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf [LEADER]: Leader has been unable to successfully communicate with peer 878235fb05a24dd48d48525ad7dbf3a5 for more than 300 seconds (300.069s)
W20260810 18:39:44.250689 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260810 18:39:44.404451 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260810 18:39:44.999022 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20260810 18:39:45.424062 31342 consensus_queue.cc:579] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf [LEADER]: Leader has been unable to successfully communicate with peer 878235fb05a24dd48d48525ad7dbf3a5 for more than 300 seconds (300.080s)
I20260810 18:39:45.674659 31305 consensus_queue.cc:579] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 [LEADER]: Leader has been unable to successfully communicate with peer 878235fb05a24dd48d48525ad7dbf3a5 for more than 300 seconds (300.283s)
W20260810 18:39:45.678737 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20260810 18:39:46.545713 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260810 18:39:46.994937 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260810 18:39:47.895299 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260810 18:39:48.352149 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:39:49.240267 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260810 18:39:49.626653 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260810 18:39:50.133270 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:39:50.341967 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260810 18:39:51.697623 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260810 18:39:51.951368 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260810 18:39:52.921319 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260810 18:39:53.855180 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:39:54.243435 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260810 18:39:54.344724 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260810 18:39:55.401087 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 18:39:55.493809 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260810 18:39:56.578935 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260810 18:39:56.780234 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260810 18:39:58.021850 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260810 18:39:58.909891 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:39:59.220492 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260810 18:39:59.452471 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260810 18:40:00.684628 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260810 18:40:00.796603 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:40:01.844903 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260810 18:40:01.866483 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260810 18:40:03.310595 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260810 18:40:04.059201 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260810 18:40:04.392087 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:40:04.416642 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260810 18:40:05.835651 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:40:05.879061 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260810 18:40:06.628825 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260810 18:40:06.905507 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260810 18:40:08.237645 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260810 18:40:09.044728 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260810 18:40:09.607152 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260810 18:40:09.888639 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:40:10.766427 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260810 18:40:11.082598 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:40:11.569666 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260810 18:40:12.352433 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260810 18:40:13.167785 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260810 18:40:13.948068 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260810 18:40:14.862668 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260810 18:40:15.248098 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:40:15.672672 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260810 18:40:16.310976 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:40:16.312000 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260810 18:40:17.558750 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260810 18:40:18.200508 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260810 18:40:18.643160 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260810 18:40:20.055717 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260810 18:40:20.563335 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:40:20.564422 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260810 18:40:21.141892 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260810 18:40:21.374871 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:40:22.503355 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260810 18:40:23.041577 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260810 18:40:23.405534 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260810 18:40:25.069008 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260810 18:40:25.515659 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260810 18:40:25.837706 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260810 18:40:25.979261 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:40:26.594043 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:40:27.556460 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260810 18:40:27.939929 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260810 18:40:28.206377 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260810 18:40:29.986495 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260810 18:40:30.334429 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260810 18:40:30.671458 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260810 18:40:31.115255 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:40:31.936833 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 18:40:32.484714 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260810 18:40:32.639304 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260810 18:40:33.069674 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260810 18:40:34.908556 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260810 18:40:35.409399 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260810 18:40:35.522436 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260810 18:40:36.604969 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:40:37.141021 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:40:37.501098 26623 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (241 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:40:37.620890 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260810 18:40:37.665047 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260810 18:40:37.752058 26231 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (241 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:40:38.290654 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260810 18:40:39.913787 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260810 18:40:40.193188 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260810 18:40:40.906277 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.25.80.62:42753) | 127.25.80.62:42753 | UNAVAILABLE
Error from 127.25.80.62:42753: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.25.80.62:42753, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
FAILED
I20260810 18:40:41.515137 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.25.80.62:42753, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:40:42.009642 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:40:42.211825 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:40:42.602051 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260810 18:40:42.829500 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260810 18:40:43.498956 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260810 18:40:45.287801 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260810 18:40:45.387051 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260810 18:40:45.838284 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260810 18:40:47.201274 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:40:47.383697 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:40:47.633176 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260810 18:40:48.152307 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260810 18:40:48.371855 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260810 18:40:50.153827 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260810 18:40:50.713198 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260810 18:40:50.841131 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260810 18:40:52.541781 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:40:52.542837 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260810 18:40:52.564055 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:40:53.307467 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260810 18:40:53.561707 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260810 18:40:54.912721 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260810 18:40:55.901053 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260810 18:40:56.050825 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260810 18:40:57.344820 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260810 18:40:57.808063 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:40:57.897243 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:40:58.432475 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260810 18:40:58.469815 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260810 18:41:00.078728 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260810 18:41:00.787189 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260810 18:41:01.091516 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260810 18:41:02.465102 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260810 18:41:02.921228 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:41:02.993710 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:41:03.170053 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260810 18:41:03.558250 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260810 18:41:04.928941 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260810 18:41:05.574952 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260810 18:41:06.394953 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260810 18:41:07.432269 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260810 18:41:08.007448 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:41:08.008004 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260810 18:41:08.090900 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260810 18:41:08.834450 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260810 18:41:09.764456 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260810 18:41:10.403704 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260810 18:41:11.345770 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260810 18:41:12.198125 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260810 18:41:13.122370 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:41:13.123416 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260810 18:41:13.202482 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:41:13.707082 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260810 18:41:14.699965 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260810 18:41:15.587682 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260810 18:41:16.113119 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260810 18:41:17.377658 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260810 18:41:18.176216 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:41:18.231571 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260810 18:41:18.278896 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:41:18.721599 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260810 18:41:19.910020 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260810 18:41:20.552731 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260810 18:41:21.390492 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260810 18:41:22.369261 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260810 18:41:23.089687 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260810 18:41:23.370194 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:41:23.519842 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:41:23.935683 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260810 18:41:24.957032 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260810 18:41:25.477665 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260810 18:41:26.420740 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260810 18:41:27.224233 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260810 18:41:28.045439 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260810 18:41:28.381019 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:41:28.558770 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:41:28.989035 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260810 18:41:29.853048 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260810 18:41:30.426340 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260810 18:41:31.582435 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260810 18:41:32.606351 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260810 18:41:32.802440 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260810 18:41:33.638172 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:41:33.756541 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:41:34.069003 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260810 18:41:35.162526 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260810 18:41:35.469849 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260810 18:41:36.420166 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260810 18:41:37.586537 26623 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (301 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:41:37.674791 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260810 18:41:37.776324 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260810 18:41:37.838594 26231 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (301 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:41:38.714311 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260810 18:41:38.937582 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:41:39.099853 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 18:41:39.856830 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260810 18:41:40.357963 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260810 18:41:41.441180 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260810 18:41:42.411602 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260810 18:41:42.775007 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.25.80.62:42753) | 127.25.80.62:42753 | UNAVAILABLE
Error from 127.25.80.62:42753: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.25.80.62:42753, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
FAILED
I20260810 18:41:42.874718 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.25.80.62:42753, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:41:43.961585 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260810 18:41:44.335744 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:41:44.336395 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:41:44.751261 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260810 18:41:45.368240 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260810 18:41:46.474427 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260810 18:41:47.248093 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260810 18:41:47.822276 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260810 18:41:49.310467 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260810 18:41:49.594342 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:41:49.595340 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260810 18:41:49.809044 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:41:50.386843 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260810 18:41:51.784941 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260810 18:41:52.065425 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260810 18:41:52.885502 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260810 18:41:54.500597 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260810 18:41:54.521783 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260810 18:41:54.859483 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:41:54.914978 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:41:55.275581 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260810 18:41:56.806207 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260810 18:41:56.911367 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260810 18:41:57.926016 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260810 18:41:59.242501 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260810 18:41:59.397166 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260810 18:41:59.983763 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:42:00.038329 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:42:00.775691 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260810 18:42:01.384147 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260810 18:42:01.954414 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260810 18:42:03.232409 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260810 18:42:03.983999 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260810 18:42:04.361972 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260810 18:42:05.067239 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:42:05.527552 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:42:05.830668 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260810 18:42:06.502022 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260810 18:42:07.090031 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260810 18:42:08.317934 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260810 18:42:08.685384 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260810 18:42:09.587584 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260810 18:42:10.297617 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:42:10.795300 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:42:10.978024 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260810 18:42:11.339524 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260810 18:42:12.285068 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260810 18:42:13.609319 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260810 18:42:13.934916 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260810 18:42:14.810329 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260810 18:42:15.450608 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:42:16.135526 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260810 18:42:16.296430 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:42:16.586169 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260810 18:42:17.416639 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260810 18:42:18.714269 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260810 18:42:19.135039 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260810 18:42:19.856912 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260810 18:42:20.706701 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:42:21.335978 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260810 18:42:21.348421 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:42:21.881198 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260810 18:42:22.392477 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260810 18:42:23.778045 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260810 18:42:24.202706 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260810 18:42:25.103696 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260810 18:42:26.012833 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:42:26.030115 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260810 18:42:26.445693 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:42:26.861943 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260810 18:42:27.638655 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260810 18:42:28.324100 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260810 18:42:29.730500 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260810 18:42:30.346719 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260810 18:42:30.822290 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260810 18:42:31.111722 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:42:31.476382 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20260810 18:42:31.990996 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260810 18:42:32.890379 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260810 18:42:33.660528 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260810 18:42:34.385842 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260810 18:42:35.206676 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260810 18:42:35.907096 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260810 18:42:36.307873 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:42:36.754685 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:42:37.016165 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260810 18:42:37.670748 26623 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (361 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:42:37.850324 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260810 18:42:37.924409 26231 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (361 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:42:38.466683 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260810 18:42:39.524457 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260810 18:42:40.421412 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260810 18:42:41.122658 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260810 18:42:41.365454 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:42:41.955665 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260810 18:42:42.026881 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:42:42.889066 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260810 18:42:43.862172 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.25.80.62:42753) | 127.25.80.62:42753 | UNAVAILABLE
Error from 127.25.80.62:42753: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.25.80.62:42753, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
FAILED
I20260810 18:42:44.138720 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.25.80.62:42753, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:42:44.538389 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260810 18:42:45.295131 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260810 18:42:46.523837 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260810 18:42:46.524595 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260810 18:42:46.976958 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260810 18:42:47.501765 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:42:47.976771 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260810 18:42:49.262107 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260810 18:42:49.424649 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260810 18:42:50.478163 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260810 18:42:51.554062 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:42:51.555084 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260810 18:42:51.913934 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260810 18:42:52.513877 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:42:53.012051 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260810 18:42:54.038410 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260810 18:42:54.412110 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260810 18:42:55.505955 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260810 18:42:56.306936 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260810 18:42:56.813545 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:42:56.814415 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260810 18:42:57.675287 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:42:58.163355 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260810 18:42:59.027015 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260810 18:42:59.172518 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260810 18:43:00.592051 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260810 18:43:01.593171 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260810 18:43:01.887588 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:43:01.888688 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260810 18:43:03.179404 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:43:03.180799 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260810 18:43:04.026450 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260810 18:43:04.328063 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260810 18:43:05.830027 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260810 18:43:06.381866 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260810 18:43:06.747153 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260810 18:43:06.913906 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:43:08.489451 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:43:08.490811 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260810 18:43:08.887761 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260810 18:43:09.191283 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260810 18:43:11.299889 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260810 18:43:11.416237 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260810 18:43:11.703064 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260810 18:43:12.088303 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:43:13.766969 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260810 18:43:14.042583 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:43:14.043856 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260810 18:43:14.324260 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260810 18:43:16.245697 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260810 18:43:16.522074 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260810 18:43:17.070884 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260810 18:43:17.296618 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:43:18.981787 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260810 18:43:18.987519 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260810 18:43:19.548777 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:43:19.551138 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260810 18:43:21.217092 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260810 18:43:21.387830 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260810 18:43:22.395236 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:43:22.396183 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260810 18:43:23.729892 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260810 18:43:24.066337 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260810 18:43:24.954779 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:43:25.068600 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260810 18:43:26.510715 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260810 18:43:26.687770 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260810 18:43:27.510208 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260810 18:43:27.682729 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260810 18:43:28.974596 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260810 18:43:29.216781 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260810 18:43:30.089576 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:43:30.405061 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260810 18:43:31.247465 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260810 18:43:31.412637 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260810 18:43:32.573251 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:43:32.824368 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260810 18:43:33.467068 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260810 18:43:34.231470 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260810 18:43:35.159881 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:43:35.376267 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260810 18:43:35.940955 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260810 18:43:36.613849 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260810 18:43:37.754112 26623 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (421 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:43:37.821192 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:43:37.822219 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260810 18:43:38.012900 26231 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (421 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:43:38.559844 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260810 18:43:39.355964 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260810 18:43:40.274421 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:43:40.460243 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260810 18:43:40.941154 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260810 18:43:41.662740 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260810 18:43:42.861497 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:43:42.862649 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260810 18:43:43.367339 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260810 18:43:44.307942 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.25.80.62:42753) | 127.25.80.62:42753 | UNAVAILABLE
Error from 127.25.80.62:42753: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.25.80.62:42753, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
FAILED
I20260810 18:43:45.441759 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.25.80.62:42753, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:43:45.512223 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260810 18:43:45.649801 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:43:45.880851 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260810 18:43:46.748775 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260810 18:43:48.264250 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:43:48.265169 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260810 18:43:48.299469 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260810 18:43:49.217940 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260810 18:43:50.645701 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260810 18:43:50.718544 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:43:50.976837 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260810 18:43:51.699438 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260810 18:43:53.127372 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260810 18:43:53.574604 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:43:53.575639 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260810 18:43:54.139976 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260810 18:43:55.598244 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260810 18:43:56.249720 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:43:56.269868 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260810 18:43:56.875030 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260810 18:43:58.101439 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260810 18:43:58.596518 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:43:58.905607 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260810 18:43:59.488159 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260810 18:44:00.740089 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260810 18:44:01.264031 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260810 18:44:01.443941 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:44:01.935496 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260810 18:44:03.181818 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260810 18:44:03.645021 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:44:03.646082 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260810 18:44:04.524039 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260810 18:44:05.688482 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260810 18:44:06.330684 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260810 18:44:06.872748 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:44:07.286799 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260810 18:44:08.191323 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260810 18:44:08.699608 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:44:08.700549 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260810 18:44:09.818619 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260810 18:44:10.444762 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260810 18:44:11.342113 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260810 18:44:12.248807 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:44:12.249900 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260810 18:44:12.830770 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260810 18:44:13.360227 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260810 18:44:13.849077 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:44:14.800477 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260810 18:44:15.249632 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260810 18:44:15.697803 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260810 18:44:17.590351 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:44:17.591617 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260810 18:44:17.722471 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260810 18:44:18.428545 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260810 18:44:18.886767 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:44:19.874742 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260810 18:44:20.542876 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260810 18:44:20.864477 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260810 18:44:22.163275 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260810 18:44:22.608793 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:44:23.045755 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260810 18:44:23.373204 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260810 18:44:23.968039 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:44:24.582103 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260810 18:44:25.764489 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260810 18:44:26.237200 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260810 18:44:27.186386 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260810 18:44:27.651647 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:44:28.345086 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260810 18:44:28.554479 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260810 18:44:29.097656 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:44:29.717296 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260810 18:44:31.080135 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260810 18:44:31.104928 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260810 18:44:32.230643 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260810 18:44:32.798139 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:44:33.360351 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260810 18:44:33.448553 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260810 18:44:34.229097 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:44:35.006807 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260810 18:44:35.659515 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260810 18:44:36.049085 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260810 18:44:37.458829 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260810 18:44:37.836874 26623 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (481 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:44:38.002578 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:44:38.101262 26231 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (481 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:44:38.168044 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260810 18:44:38.336836 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260810 18:44:39.360113 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:44:39.984346 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260810 18:44:40.631049 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260810 18:44:40.655230 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260810 18:44:42.632581 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260810 18:44:43.133121 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:44:43.192288 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260810 18:44:43.416344 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260810 18:44:44.555408 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:44:45.234194 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260810 18:44:45.603999 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260810 18:44:45.803282 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.25.80.62:42753) | 127.25.80.62:42753 | UNAVAILABLE
Error from 127.25.80.62:42753: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.25.80.62:42753, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
FAILED
I20260810 18:44:46.701208 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.25.80.62:42753, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:44:47.885842 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260810 18:44:48.246301 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260810 18:44:48.464177 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:44:48.547111 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260810 18:44:49.616485 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:44:50.499181 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260810 18:44:50.541419 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260810 18:44:51.070218 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260810 18:44:52.957283 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260810 18:44:53.122491 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260810 18:44:53.479772 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260810 18:44:53.518297 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:44:54.736667 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:44:55.192742 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260810 18:44:55.712316 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260810 18:44:56.071712 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260810 18:44:57.554427 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260810 18:44:58.621523 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260810 18:44:58.685645 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260810 18:44:58.778916 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:44:59.810824 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260810 18:45:00.118014 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260810 18:45:00.894737 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260810 18:45:01.246197 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260810 18:45:02.894716 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260810 18:45:03.519378 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260810 18:45:03.533922 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260810 18:45:03.801004 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:45:05.134203 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:45:05.372313 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260810 18:45:05.995164 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260810 18:45:06.042487 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260810 18:45:07.684244 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260810 18:45:08.525584 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260810 18:45:08.676124 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260810 18:45:09.306509 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:45:10.181628 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:45:10.278520 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260810 18:45:11.183826 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260810 18:45:11.281252 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260810 18:45:12.672776 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260810 18:45:13.705387 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260810 18:45:13.931852 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260810 18:45:14.687701 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:45:15.155948 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260810 18:45:15.328650 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260810 18:45:16.260219 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260810 18:45:16.400118 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260810 18:45:17.639400 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260810 18:45:18.744544 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260810 18:45:19.074656 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260810 18:45:19.707338 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:45:20.180718 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260810 18:45:20.770217 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 18:45:21.210664 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260810 18:45:21.303072 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260810 18:45:22.828436 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260810 18:45:23.778273 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260810 18:45:23.817690 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260810 18:45:24.737879 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:45:25.179617 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260810 18:45:26.167310 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 18:45:26.168366 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260810 18:45:26.214900 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260810 18:45:27.592144 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260810 18:45:28.738092 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260810 18:45:28.783351 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260810 18:45:30.181116 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:45:30.182315 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260810 18:45:30.985317 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260810 18:45:31.127317 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260810 18:45:31.435545 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 18:45:32.638517 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260810 18:45:33.432361 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260810 18:45:33.610297 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260810 18:45:35.261480 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:45:35.262859 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260810 18:45:35.979699 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260810 18:45:36.113797 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260810 18:45:36.517771 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:45:37.700299 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260810 18:45:37.918893 26623 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (541 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:45:38.187139 26231 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (541 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:45:38.570765 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260810 18:45:38.820029 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260810 18:45:40.300709 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:45:40.302132 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260810 18:45:40.919126 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260810 18:45:41.100518 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260810 18:45:41.570853 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:45:42.943117 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260810 18:45:43.463362 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260810 18:45:43.799376 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260810 18:45:45.590633 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:45:45.591660 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260810 18:45:46.028237 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260810 18:45:46.484967 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260810 18:45:46.626557 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260810 18:45:47.958012 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.25.80.62:42753) | 127.25.80.62:42753 | UNAVAILABLE
Error from 127.25.80.62:42753: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.25.80.62:42753, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
FAILED
I20260810 18:45:48.013923 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.25.80.62:42753, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:45:48.749656 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260810 18:45:49.214164 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260810 18:45:50.290617 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260810 18:45:50.756712 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:45:50.989053 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260810 18:45:51.735314 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:45:51.736351 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260810 18:45:52.672084 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260810 18:45:53.399664 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260810 18:45:54.336632 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260810 18:45:55.204500 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260810 18:45:55.815009 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:45:55.931641 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260810 18:45:56.800487 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:45:56.801550 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260810 18:45:57.593709 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260810 18:45:58.364574 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260810 18:45:59.144603 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260810 18:46:00.159015 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260810 18:46:00.795945 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260810 18:46:01.239475 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:46:01.658787 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260810 18:46:01.811944 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:46:02.745023 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260810 18:46:03.422116 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260810 18:46:04.160409 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260810 18:46:05.174249 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260810 18:46:05.534092 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260810 18:46:06.581539 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260810 18:46:06.727000 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:46:07.101821 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:46:07.833194 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260810 18:46:08.344441 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260810 18:46:09.041899 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260810 18:46:10.692821 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260810 18:46:10.701048 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260810 18:46:11.617465 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260810 18:46:12.171015 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:46:12.216295 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:46:13.129158 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260810 18:46:13.157935 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260810 18:46:13.963547 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260810 18:46:15.395740 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260810 18:46:15.411573 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260810 18:46:16.197456 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260810 18:46:17.242609 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:46:17.442094 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 18:46:17.764353 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260810 18:46:17.934998 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260810 18:46:18.388622 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260810 18:46:20.159475 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260810 18:46:20.721469 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260810 18:46:20.790305 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260810 18:46:22.317022 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:46:22.613889 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:46:22.907027 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260810 18:46:23.221623 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260810 18:46:23.250273 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260810 18:46:25.321156 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260810 18:46:25.658767 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260810 18:46:25.700115 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260810 18:46:27.440769 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:46:27.441951 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260810 18:46:28.012439 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 18:46:28.013603 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260810 18:46:28.047636 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260810 18:46:30.016249 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260810 18:46:30.530284 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260810 18:46:30.778147 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260810 18:46:32.793844 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:46:32.795221 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260810 18:46:33.200507 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:46:33.201892 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260810 18:46:33.213409 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260810 18:46:35.092097 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260810 18:46:35.677286 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260810 18:46:35.751535 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260810 18:46:37.314082 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260810 18:46:38.002344 26623 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (601 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:46:38.093907 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260810 18:46:38.138537 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20260810 18:46:38.252149 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:46:38.253436 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260810 18:46:38.272596 26231 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (601 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:46:39.675782 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260810 18:46:40.713522 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260810 18:46:40.786190 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260810 18:46:42.161255 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260810 18:46:43.052160 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260810 18:46:43.181923 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:46:43.223510 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260810 18:46:43.465482 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:46:44.605126 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260810 18:46:45.276582 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260810 18:46:45.731737 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260810 18:46:47.156632 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260810 18:46:47.836123 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260810 18:46:48.267656 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260810 18:46:48.649287 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:46:48.760278 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.25.80.62:42753) | 127.25.80.62:42753 | UNAVAILABLE
Error from 127.25.80.62:42753: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.25.80.62:42753, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
FAILED
I20260810 18:46:49.342201 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.25.80.62:42753, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:46:49.699911 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260810 18:46:50.212317 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260810 18:46:50.583654 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260810 18:46:52.039019 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260810 18:46:52.868212 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260810 18:46:53.215382 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260810 18:46:53.659710 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:46:54.072175 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:46:54.591660 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260810 18:46:55.171703 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260810 18:46:55.662204 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260810 18:46:57.155685 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260810 18:46:57.824136 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260810 18:46:58.162470 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260810 18:46:59.005874 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:46:59.112887 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:46:59.562961 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260810 18:47:00.497442 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260810 18:47:00.626708 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260810 18:47:02.020834 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260810 18:47:02.985668 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260810 18:47:03.373631 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260810 18:47:04.453824 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:47:04.515216 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:47:04.516875 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260810 18:47:05.608896 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260810 18:47:06.191861 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260810 18:47:06.763170 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260810 18:47:08.206424 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260810 18:47:08.750996 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260810 18:47:09.165673 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260810 18:47:09.564182 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:47:09.634639 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:47:10.768829 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260810 18:47:11.083799 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260810 18:47:11.456267 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260810 18:47:13.376672 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260810 18:47:13.583213 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260810 18:47:14.058513 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260810 18:47:14.770337 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:47:15.137951 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:47:15.876396 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260810 18:47:16.022919 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260810 18:47:16.909384 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260810 18:47:18.588387 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260810 18:47:18.702360 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260810 18:47:19.249480 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260810 18:47:19.824579 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:47:20.336151 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:47:20.830287 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260810 18:47:21.100055 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260810 18:47:21.967486 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260810 18:47:23.228562 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260810 18:47:23.554311 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260810 18:47:24.257488 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260810 18:47:24.993176 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:47:25.475811 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260810 18:47:25.589254 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:47:26.247583 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260810 18:47:26.451269 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260810 18:47:28.002086 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260810 18:47:28.787034 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260810 18:47:28.902495 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260810 18:47:30.260447 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:47:30.637939 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260810 18:47:30.905695 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:47:31.166441 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260810 18:47:31.492241 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260810 18:47:33.126753 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260810 18:47:33.667618 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260810 18:47:33.832036 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260810 18:47:35.526208 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 18:47:35.527314 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260810 18:47:35.931720 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:47:36.328194 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260810 18:47:36.481590 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260810 18:47:37.892084 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260810 18:47:38.089396 26623 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (661 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:47:38.360190 26231 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (661 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:47:38.991160 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260810 18:47:39.193807 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260810 18:47:40.576438 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:47:40.577484 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260810 18:47:41.021430 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:47:41.176052 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260810 18:47:41.549417 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260810 18:47:43.102068 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260810 18:47:43.534616 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260810 18:47:44.027894 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260810 18:47:45.738129 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:47:45.739132 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260810 18:47:45.760820 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260810 18:47:46.375874 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:47:46.376868 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260810 18:47:48.387097 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260810 18:47:48.397329 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260810 18:47:48.716820 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.25.80.62:42753) | 127.25.80.62:42753 | UNAVAILABLE
Error from 127.25.80.62:42753: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.25.80.62:42753, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
FAILED
I20260810 18:47:50.682754 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.25.80.62:42753, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:47:50.691263 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260810 18:47:51.066113 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:47:51.067106 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260810 18:47:51.214562 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260810 18:47:51.690219 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:47:53.212378 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260810 18:47:53.496914 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260810 18:47:53.590571 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260810 18:47:55.806028 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260810 18:47:55.817543 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260810 18:47:55.861204 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260810 18:47:56.184068 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:47:57.036471 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:47:58.273548 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260810 18:47:58.423326 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260810 18:47:58.428086 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260810 18:48:00.787678 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260810 18:48:01.062713 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260810 18:48:01.149963 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260810 18:48:01.285609 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:48:02.039512 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:48:03.271268 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260810 18:48:03.277842 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260810 18:48:03.622938 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260810 18:48:05.880843 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260810 18:48:06.097446 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260810 18:48:06.181303 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260810 18:48:06.627401 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:48:07.336115 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:48:08.250964 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260810 18:48:08.438557 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260810 18:48:08.458166 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260810 18:48:10.830248 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260810 18:48:11.017026 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260810 18:48:11.058604 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260810 18:48:11.736339 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 18:48:12.798755 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:48:13.128695 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260810 18:48:13.421470 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260810 18:48:13.955902 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260810 18:48:15.595965 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260810 18:48:15.868794 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260810 18:48:16.572381 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260810 18:48:16.894197 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:48:17.988283 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260810 18:48:18.218603 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:48:18.380292 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260810 18:48:19.289707 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260810 18:48:20.570389 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260810 18:48:20.622164 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260810 18:48:21.681195 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260810 18:48:22.054379 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:48:23.004845 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260810 18:48:23.159837 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260810 18:48:23.579835 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:48:24.073164 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260810 18:48:25.470592 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260810 18:48:25.546254 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260810 18:48:26.495340 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260810 18:48:27.279870 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 18:48:27.752445 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260810 18:48:27.883455 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260810 18:48:28.594616 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:48:29.072081 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260810 18:48:30.183413 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260810 18:48:30.228096 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260810 18:48:31.548446 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260810 18:48:32.290899 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:48:32.749125 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260810 18:48:32.833148 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260810 18:48:33.892501 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:48:33.893736 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260810 18:48:35.454245 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260810 18:48:35.454944 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260810 18:48:36.279171 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260810 18:48:37.568676 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260810 18:48:38.033756 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260810 18:48:38.167943 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260810 18:48:38.172988 26623 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (721 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:48:38.448799 26231 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (721 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:48:38.725299 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260810 18:48:39.323990 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:48:40.454648 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260810 18:48:40.834443 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260810 18:48:41.298388 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260810 18:48:42.913614 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:48:43.040477 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260810 18:48:43.345494 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260810 18:48:43.678244 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260810 18:48:44.646456 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:48:45.704440 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260810 18:48:45.903254 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260810 18:48:46.018476 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260810 18:48:47.921533 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:48:47.922695 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260810 18:48:48.442818 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260810 18:48:48.464475 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260810 18:48:50.025465 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:48:50.423410 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260810 18:48:50.924597 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260810 18:48:51.015585 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.25.80.62:42753) | 127.25.80.62:42753 | UNAVAILABLE
Error from 127.25.80.62:42753: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.25.80.62:42753, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
FAILED
I20260810 18:48:51.956678 25920 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.25.80.62:42753, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
W20260810 18:48:52.922864 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:48:52.996249 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260810 18:48:53.300992 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260810 18:48:53.580164 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260810 18:48:55.170382 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:48:55.564939 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260810 18:48:55.834800 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260810 18:48:56.203758 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260810 18:48:58.144320 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:48:58.145542 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260810 18:48:58.156558 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260810 18:48:58.678320 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260810 18:49:00.652794 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260810 18:49:00.682355 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:49:00.711843 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260810 18:49:01.295965 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260810 18:49:03.226889 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 18:49:03.227842 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260810 18:49:03.279311 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260810 18:49:03.966462 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260810 18:49:05.485252 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260810 18:49:05.797195 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
************************ BEGIN STACKS **************************
W20260810 18:49:06.122948 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
[New LWP 25923]
[New LWP 25925]
[New LWP 25926]
[New LWP 26630]
[New LWP 26631]
[New LWP 26632]
[New LWP 26633]
[New LWP 400]
[New LWP 401]
[New LWP 402]
[New LWP 403]
[New LWP 408]
[New LWP 409]
[New LWP 410]
[New LWP 412]
[New LWP 413]
W20260810 18:49:06.587733 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260810 18:49:07.676234 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260810 18:49:08.215181 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260810 18:49:08.694031 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20260810 18:49:09.125840 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260810 18:49:10.252816 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260810 18:49:10.550472 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260810 18:49:11.387821 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:49:11.388887 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260810 18:49:12.874454 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260810 18:49:12.982992 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260810 18:49:13.877727 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:49:14.149307 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260810 18:49:15.356192 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260810 18:49:15.529800 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260810 18:49:16.814642 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:49:16.815950 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260810 18:49:17.830525 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260810 18:49:18.352686 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260810 18:49:18.914745 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260810 18:49:19.435326 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260810 18:49:20.498626 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260810 18:49:20.864058 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260810 18:49:22.148057 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:49:22.150477 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260810 18:49:23.070108 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260810 18:49:23.374475 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260810 18:49:24.170706 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 18:49:24.534185 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260810 18:49:25.764873 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260810 18:49:25.862905 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260810 18:49:27.078857 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260810 18:49:27.548977 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:49:28.189247 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260810 18:49:28.305662 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260810 18:49:29.206357 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
W20260810 18:49:29.725180 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260810 18:49:30.537048 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260810 18:49:30.794750 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260810 18:49:32.054324 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260810 18:49:32.667344 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 18:49:32.925872 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
W20260810 18:49:33.029220 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
0x00007f4f89774ad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
Id Target Id Frame
* 1 Thread 0x7f4f81c75d40 (LWP 25920) "all_types-itest" 0x00007f4f89774ad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
2 Thread 0x7f4f7d7bd700 (LWP 25923) "all_types-itest" 0x00007f4f8977932a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
3 Thread 0x7f4f7c1f4700 (LWP 25925) "kernel-watcher-" 0x00007f4f89774fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
4 Thread 0x7f4f7b9f3700 (LWP 25926) "rpc reactor-259" 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
5 Thread 0x7f4f7b1f2700 (LWP 26630) "rpc reactor-266" 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
6 Thread 0x7f4f7a7ea700 (LWP 26631) "rpc reactor-266" 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
7 Thread 0x7f4f79fe9700 (LWP 26632) "rpc reactor-266" 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
8 Thread 0x7f4f797e8700 (LWP 26633) "rpc reactor-266" 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
9 Thread 0x7f4f77d92700 (LWP 400) "rpc reactor-400" 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
10 Thread 0x7f4f740c5700 (LWP 401) "rpc reactor-401" 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
11 Thread 0x7f4f738c4700 (LWP 402) "rpc reactor-402" 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
12 Thread 0x7f4f7658f700 (LWP 403) "rpc reactor-403" 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
13 Thread 0x7f4f77591700 (LWP 408) "rpc reactor-408" 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
14 Thread 0x7f4f76d90700 (LWP 409) "rpc reactor-409" 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
15 Thread 0x7f4f75d8e700 (LWP 410) "rpc reactor-410" 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
16 Thread 0x7f4f7558d700 (LWP 412) "rpc reactor-412" 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
17 Thread 0x7f4f74d8c700 (LWP 413) "client-negotiat" 0x00007f4f89774fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
Thread 17 (Thread 0x7f4f74d8c700 (LWP 413)):
#0 0x00007f4f89774fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f4f8695f8d9 in kudu::ConditionVariable::WaitFor (this=0x7f4f89774fb9 <pthread_cond_timedwait@@GLIBC_2.3.2+649>, delta=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/condition_variable.cc:121
W20260810 18:49:34.403688 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 20 similar messages]
#2 0x00007f4f86c0fe0a in kudu::ThreadPool::DispatchThread (this=0x6160004efc80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:721
#3 0x00007f4f86c1391c in kudu::ThreadPool::CreateThread()::$_2::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:849
W20260810 18:49:34.628494 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
#4 0x00007f4f86c13799 in std::_Function_handler<void (), kudu::ThreadPool::CreateThread()::$_2>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#5 0x00007f4f9b16abbd in std::function<void ()>::operator()() const (this=0x6110001a7768) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#6 0x00007f4f86beeeaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#7 0x00007f4f8976e6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#8 0x00007f4f82a4861f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 16 (Thread 0x7f4f7558d700 (LWP 412)):
#0 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f4f84b37bda in epoll_poll (loop=0x619000172a80, timeout=0.098926542559638619) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f4f84b3b228 in ev_run (loop=0x619000172a80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4f8d1d3e31 in ev::loop_ref::run (this=this@entry=0x614000099288, flags=7264000, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4f8d1cc326 in kudu::rpc::ReactorThread::RunThread (this=0x614000099280) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4f8d1d13ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4f8d1d1229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f4f9b16abbd in std::function<void ()>::operator()() const (this=0x6110008562a8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f4f86beeeaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f4f8976e6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
W20260810 18:49:35.388513 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
#10 0x00007f4f82a4861f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 15 (Thread 0x7f4f75d8e700 (LWP 410)):
#0 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f4f84b37bda in epoll_poll (loop=0x619000175780, timeout=0.099926299648359418) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f4f84b3b228 in ev_run (loop=0x619000175780, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4f8d1d3e31 in ev::loop_ref::run (this=this@entry=0x614000098c88, flags=7263104, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4f8d1cc326 in kudu::rpc::ReactorThread::RunThread (this=0x614000098c80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4f8d1d13ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4f8d1d1229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f4f9b16abbd in std::function<void ()>::operator()() const (this=0x611000856168) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f4f86beeeaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f4f8976e6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f4f82a4861f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 14 (Thread 0x7f4f76d90700 (LWP 409)):
#0 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f4f84b37bda in epoll_poll (loop=0x619000172f80, timeout=0.099950287491083145) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f4f84b3b228 in ev_run (loop=0x619000172f80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4f8d1d3e31 in ev::loop_ref::run (this=this@entry=0x614000099488, flags=7262208, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4f8d1cc326 in kudu::rpc::ReactorThread::RunThread (this=0x614000099480) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4f8d1d13ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4f8d1d1229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f4f9b16abbd in std::function<void ()>::operator()() const (this=0x611000856028) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f4f86beeeaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f4f8976e6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f4f82a4861f in clone () from /lib/x86_64-linux-gnu/libc.so.6
W20260810 18:49:35.424609 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
Thread 13 (Thread 0x7f4f77591700 (LWP 408)):
#0 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f4f84b37bda in epoll_poll (loop=0x619000175c80, timeout=0.035753704607486725) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f4f84b3b228 in ev_run (loop=0x619000175c80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4f8d1d3e31 in ev::loop_ref::run (this=this@entry=0x614000099688, flags=7261312, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4f8d1cc326 in kudu::rpc::ReactorThread::RunThread (this=0x614000099680) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4f8d1d13ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4f8d1d1229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f4f9b16abbd in std::function<void ()>::operator()() const (this=0x611000855ee8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f4f86beeeaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f4f8976e6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f4f82a4861f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 12 (Thread 0x7f4f7658f700 (LWP 403)):
#0 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f4f84b37bda in epoll_poll (loop=0x61900027ab80, timeout=0.098830696661025286) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f4f84b3b228 in ev_run (loop=0x61900027ab80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4f8d1d3e31 in ev::loop_ref::run (this=this@entry=0x61400048d288, flags=7258624, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4f8d1cc326 in kudu::rpc::ReactorThread::RunThread (this=0x61400048d280) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4f8d1d13ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4f8d1d1229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f4f9b16abbd in std::function<void ()>::operator()() const (this=0x611000855268) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f4f86beeeaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f4f8976e6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f4f82a4861f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 11 (Thread 0x7f4f738c4700 (LWP 402)):
#0 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f4f84b37bda in epoll_poll (loop=0x61900027a180, timeout=0.09882962959818542) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f4f84b3b228 in ev_run (loop=0x61900027a180, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4f8d1d3e31 in ev::loop_ref::run (this=this@entry=0x61400048d088, flags=7257728, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4f8d1cc326 in kudu::rpc::ReactorThread::RunThread (this=0x61400048d080) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4f8d1d13ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4f8d1d1229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f4f9b16abbd in std::function<void ()>::operator()() const (this=0x611000855128) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f4f86beeeaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f4f8976e6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f4f82a4861f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 10 (Thread 0x7f4f740c5700 (LWP 401)):
#0 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f4f84b37bda in epoll_poll (loop=0x61900027a680, timeout=0.099978606449440122) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f4f84b3b228 in ev_run (loop=0x61900027a680, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4f8d1d3e31 in ev::loop_ref::run (this=this@entry=0x61400048ce88, flags=7256832, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4f8d1cc326 in kudu::rpc::ReactorThread::RunThread (this=0x61400048ce80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4f8d1d13ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4f8d1d1229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f4f9b16abbd in std::function<void ()>::operator()() const (this=0x611000854fe8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f4f86beeeaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f4f8976e6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f4f82a4861f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 9 (Thread 0x7f4f77d92700 (LWP 400)):
#0 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f4f84b37bda in epoll_poll (loop=0x61900027dd80, timeout=0.099635737482458353) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f4f84b3b228 in ev_run (loop=0x61900027dd80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4f8d1d3e31 in ev::loop_ref::run (this=this@entry=0x61400048cc88, flags=7255936, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4f8d1cc326 in kudu::rpc::ReactorThread::RunThread (this=0x61400048cc80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4f8d1d13ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4f8d1d1229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f4f9b16abbd in std::function<void ()>::operator()() const (this=0x611000854ea8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f4f86beeeaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f4f8976e6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f4f82a4861f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 8 (Thread 0x7f4f797e8700 (LWP 26633)):
#0 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f4f84b37bda in epoll_poll (loop=0x619000055a80, timeout=0.099038817221298814) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f4f84b3b228 in ev_run (loop=0x619000055a80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4f8d1d3e31 in ev::loop_ref::run (this=this@entry=0x614000004288, flags=16256, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4f8d1cc326 in kudu::rpc::ReactorThread::RunThread (this=0x614000004280) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4f8d1d13ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4f8d1d1229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f4f9b16abbd in std::function<void ()>::operator()() const (this=0x611000059168) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f4f86beeeaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f4f8976e6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f4f82a4861f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 7 (Thread 0x7f4f79fe9700 (LWP 26632)):
#0 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f4f84b37bda in epoll_poll (loop=0x619000055580, timeout=0.099015661049634218) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f4f84b3b228 in ev_run (loop=0x619000055580, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4f8d1d3e31 in ev::loop_ref::run (this=this@entry=0x614000004088, flags=15360, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4f8d1cc326 in kudu::rpc::ReactorThread::RunThread (this=0x614000004080) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4f8d1d13ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4f8d1d1229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f4f9b16abbd in std::function<void ()>::operator()() const (this=0x611000059028) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f4f86beeeaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f4f8976e6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f4f82a4861f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 6 (Thread 0x7f4f7a7ea700 (LWP 26631)):
#0 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f4f84b37bda in epoll_poll (loop=0x619000055080, timeout=0.099862655159085989) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f4f84b3b228 in ev_run (loop=0x619000055080, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4f8d1d3e31 in ev::loop_ref::run (this=this@entry=0x614000003e88, flags=14464, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4f8d1cc326 in kudu::rpc::ReactorThread::RunThread (this=0x614000003e80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4f8d1d13ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4f8d1d1229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f4f9b16abbd in std::function<void ()>::operator()() const (this=0x611000058ee8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f4f86beeeaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f4f8976e6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f4f82a4861f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 5 (Thread 0x7f4f7b1f2700 (LWP 26630)):
#0 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f4f84b37bda in epoll_poll (loop=0x619000054b80, timeout=0.099049804033711553) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f4f84b3b228 in ev_run (loop=0x619000054b80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4f8d1d3e31 in ev::loop_ref::run (this=this@entry=0x614000003c88, flags=13568, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4f8d1cc326 in kudu::rpc::ReactorThread::RunThread (this=0x614000003c80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4f8d1d13ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4f8d1d1229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f4f9b16abbd in std::function<void ()>::operator()() const (this=0x611000058da8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f4f86beeeaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f4f8976e6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f4f82a4861f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 4 (Thread 0x7f4f7b9f3700 (LWP 25926)):
#0 0x00007f4f82a48947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f4f84b37bda in epoll_poll (loop=0x619000052880, timeout=0.099467844935134053) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f4f84b3b228 in ev_run (loop=0x619000052880, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4f8d1d3e31 in ev::loop_ref::run (this=this@entry=0x614000003488, flags=9088, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4f8d1cc326 in kudu::rpc::ReactorThread::RunThread (this=0x614000003480) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4f8d1d13ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4f8d1d1229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f4f9b16abbd in std::function<void ()>::operator()() const (this=0x611000051fa8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f4f86beeeaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f4f8976e6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f4f82a4861f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 3 (Thread 0x7f4f7c1f4700 (LWP 25925)):
#0 0x00007f4f89774fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f4f8695f50e in kudu::ConditionVariable::WaitUntil (this=0x612000009118, until=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/condition_variable.cc:87
#2 0x00007f4f9a7b202c in kudu::CountDownLatch::WaitUntil (this=<optimized out>, when=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/countdown_latch.h:89
#3 0x00007f4f9a7aa4aa in kudu::CountDownLatch::WaitFor (this=0x612000009140, delta=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/countdown_latch.h:99
#4 0x00007f4f86a9545f in kudu::KernelStackWatchdog::RunThread (this=0x612000009040) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/kernel_stack_watchdog.cc:134
#5 0x00007f4f86a9651c in kudu::KernelStackWatchdog::KernelStackWatchdog()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/kernel_stack_watchdog.cc:71
#6 0x00007f4f86a96399 in std::_Function_handler<void (), kudu::KernelStackWatchdog::KernelStackWatchdog()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f4f9b16abbd in std::function<void ()>::operator()() const (this=0x611000052228) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f4f86beeeaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f4f8976e6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f4f82a4861f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 2 (Thread 0x7f4f7d7bd700 (LWP 25923)):
#0 0x00007f4f8977932a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000039b4fb in __interceptor_waitpid (pid=<optimized out>, status=0x7f4f7d7bb8b0, options=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2434
#2 0x00007f4f86bd9796 in kudu::Subprocess::DoWait (this=<optimized out>, wait_status=<optimized out>, mode=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/subprocess.cc:817
#3 0x00007f4f86bd94c5 in kudu::Subprocess::Wait (this=0x7f4f7d7bb8b0, wait_status=0x0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/subprocess.cc:544
#4 0x00007f4f86a6f2be in kudu::PstackWatcher::RunStackDump (argv=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:258
#5 0x00007f4f86a6e9a7 in kudu::PstackWatcher::RunGdbStackDump (pid=<optimized out>, flags=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:218
#6 0x00007f4f86a6de96 in kudu::PstackWatcher::DumpPidStacks (pid=<optimized out>, flags=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:169
#7 0x00007f4f86a6c417 in kudu::PstackWatcher::DumpStacks (flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:162
#8 0x00000000005386f9 in kudu::CreateAndStartTimeoutThread()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:60
#9 0x0000000000538583 in _ZSt13__invoke_implIvZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0JEET_St14__invoke_otherOT0_DpOT1_ (__f=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:60
#10 0x0000000000538503 in _ZSt8__invokeIZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS3_DpOS4_ (__fn=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:95
#11 0x00000000005384d9 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> >::_M_invoke<0ul> (this=this@entry=0x602000029c18) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:244
#12 0x0000000000538499 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> >::operator()() (this=this@entry=0x602000029c18) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:253
#13 0x000000000053839b in std::thread::_State_impl<std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> > >::_M_run() (this=<optimized out>) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:196
#14 0x00007f4f8e0b5a4f in execute_native_thread_routine () from /tmp/dist-test-taskJBH3sd/build/asan/bin/../../../thirdparty/installed/uninstrumented/lib/librocksdb.so.7
#15 0x00007f4f8976e6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#16 0x00007f4f82a4861f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 1 (Thread 0x7f4f81c75d40 (LWP 25920)):
#0 0x00007f4f89774ad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f4f8695f1e7 in kudu::ConditionVariable::Wait (this=0x60e0001bb470) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/condition_variable.cc:55
#2 0x00007f4f9b8211b9 in kudu::CountDownLatch::Wait (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/countdown_latch.h:79
W20260810 18:49:36.910475 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
#3 0x00007f4f9af17f4b in kudu::Synchronizer::Wait (this=0x7ffe3a2bc680) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/async_util.h:58
#4 0x00007f4f941b8ef3 in kudu::client::KuduClient::Data::ConnectToCluster (this=0x615000156380, client=<optimized out>, deadline=..., creds_policy=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/client/client-internal.cc:815
W20260810 18:49:37.696008 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
W20260810 18:49:37.907203 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
#5 0x00007f4f94138901 in kudu::client::KuduClientBuilder::Build (this=0x7ffe3a2bcf80, client=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/client/client.cc:469
W20260810 18:49:38.063333 26079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 18:49:38.261868 26623 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (781 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
#6 0x00007f4f94adeeac in kudu::tools::CreateKuduClient (master_addresses=..., client=<optimized out>, can_see_all_replicas=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/tool_action_common.cc:738
W20260810 18:49:38.541656 26231 heartbeater.cc:644] Failed to heartbeat to 127.25.80.62:42753 (781 consecutive failures): Network error: Failed to ping master at 127.25.80.62:42753: Client connection negotiation failed: client connection to 127.25.80.62:42753: connect: Connection refused (error 111)
#7 0x00007f4f9a7d7c15 in kudu::tools::RemoteKsckCluster::Connect (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck_remote.cc:528
#8 0x00007f4f9a74b87e in kudu::tools::Ksck::CheckClusterRunning (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck.cc:460
#9 0x00007f4f9a74d83f in kudu::tools::Ksck::Run (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck.cc:595
#10 0x00007f4f9a75213d in kudu::tools::Ksck::RunAndPrintResults (this=0x619000273390) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck.cc:801
#11 0x00007f4f9b7c9dd3 in kudu::ClusterVerifier::RunKsck (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/cluster_verifier.cc:113
#12 0x00007f4f9b7c9556 in kudu::ClusterVerifier::CheckCluster (this=0x7ffe3a2be3a0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/cluster_verifier.cc:78
W20260810 18:49:39.677224 26079 consensus_peers.cc:487] T 58bd2e2d08644a7883215728907d24b4 P 31cbb92d24564804aebba44e82409852 -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
W20260810 18:49:39.776247 26473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111) [suppressed 21 similar messages]
#13 0x0000000000461258 in kudu::client::AllTypesItest<kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)3> > >::RunTest(std::function<kudu::Status (kudu::client::KuduScanner*)> const&, std::function<void (kudu::client::KuduScanBatch const&, int, int*)> const&) (this=0x60c00000d000, scanner_setup=..., verifier=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/all_types-itest.cc:622
#14 0x000000000045ff4d in kudu::client::AllTypesItest_TestAllKeyTypes_Test<kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)3> > >::TestBody (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/all_types-itest.cc:676
W20260810 18:49:40.208618 26473 consensus_peers.cc:487] T 35ffbeab79c14b74b8d98a6fad1fbfa9 P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
#15 0x00007f4f87448bfe in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (location=0x7f4f8744ea28 "the test body", method=<optimized out>, object=0x60c00000d000) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest-internal-inl.h:831
#16 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0x60c00000d000, method=<optimized out>, location=location@entry=0x7f4f8744ea28 "the test body") at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#17 0x00007f4f8743d5db in testing::Test::Run (this=0x60c00000d000) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2674
#18 testing::Test::Run (this=0x60c00000d000) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2664
#19 0x00007f4f8743d795 in testing::TestInfo::Run (this=0x611000045640) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853
#20 0x00007f4f8743dcdf in testing::TestSuite::Run (this=0x611000045780) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012
#21 testing::TestSuite::Run (this=0x611000045780) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2985
#22 0x00007f4f8743e29f in testing::internal::UnitTestImpl::RunAllTests() () at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest-internal-inl.h:600
#23 0x00007f4f8744913e in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x7f4f874505a8 "auxiliary test code (environments or event listeners)", method=<optimized out>, object=0x61600000b480) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest-internal-inl.h:831
#24 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x61600000b480, method=<optimized out>, location=location@entry=0x7f4f874505a8 "auxiliary test code (environments or event listeners)") at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#25 0x00007f4f8743d83d in testing::UnitTest::Run() () at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/include/gtest/gtest.h:1246
#26 0x0000000000539482 in RUN_ALL_TESTS () at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/gtest/gtest.h:2293
#27 0x0000000000537992 in main (argc=<optimized out>, argv=0x7ffe3a2bec28) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:109
W20260810 18:49:40.435946 26473 consensus_peers.cc:487] T cfeba2dcf4954505b96a807ac76f55ce P 5f58fd4898774f4db897e97cc1598adf -> Peer 878235fb05a24dd48d48525ad7dbf3a5 (127.25.80.2:44027): Couldn't send request to peer 878235fb05a24dd48d48525ad7dbf3a5. Status: Network error: Client connection negotiation failed: client connection to 127.25.80.2:44027: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
************************* END STACKS ***************************
F20260810 18:49:40.506515 25923 test_main.cc:63] Maximum unit test time exceeded (870 sec)
*** Check failure stack trace: ***
*** Aborted at 1786387780 (unix time) try "date -d @1786387780" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800006540) received by PID 25920 (TID 0x7f4f7d7bd700) from PID 25920; stack trace: ***
@ 0x7f4f85637a46 google::(anonymous namespace)::FailureSignalHandler() at ??:0
@ 0x7f4f89779980 (unknown) at ??:0
@ 0x7f4f82965e87 gsignal at ??:0
@ 0x7f4f829677f1 abort at ??:0
@ 0x7f4f856299ad google::LogMessage::Fail() at ??:0
@ 0x7f4f8562c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f4f856294de google::LogMessage::Flush() at ??:0
@ 0x7f4f8562aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x538891 kudu::CreateAndStartTimeoutThread()::$_0::operator()() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:63
@ 0x538583 std::__invoke_impl<>() at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:60
@ 0x538503 std::__invoke<>() at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:95
@ 0x5384d9 _ZNSt6thread8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0EEE9_M_invokeIJLm0EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:244
@ 0x538499 std::thread::_Invoker<>::operator()() at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:253
@ 0x53839b std::thread::_State_impl<>::_M_run() at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:196
@ 0x7f4f8e0b5a4f execute_native_thread_routine at ??:0
@ 0x7f4f8976e6db start_thread at ??:0
@ 0x7f4f82a4861f clone at ??:0