Note: This is test shard 4 of 8.
[==========] Running 3 tests from 3 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AllTypesItest/1, where TypeParam = kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)3> >
[ RUN ] AllTypesItest/1.TestTimestampPadding
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260812 11:10:48.217293 16990 test_util.cc:268] Using random seed: -1548226920
2026-08-12T11:10:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:10:48Z Disabled control of system clock
I20260812 11:10:48.270100 16990 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknusDu0/build/asan/bin/kudu
/tmp/dist-test-tasknusDu0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.151.190:43065
--webserver_interface=127.16.151.190
--webserver_port=0
--builtin_ntp_servers=127.16.151.148:37557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.151.190:43065 with env {}
W20260812 11:10:48.700114 17003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:10:48.700546 17003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:10:48.700683 17003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:10:48.709887 17003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:10:48.710000 17003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:10:48.710043 17003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:10:48.710161 17003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:10:48.721294 17003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.151.148:37557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.151.190:43065
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.151.190:43065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.151.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:10:48.723383 17003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:10:48.726068 17003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:10:48.750170 17008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:10:48.750008 17014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:10:48.749830 17009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:10:48.751602 17003 server_base.cc:1034] running on GCE node
I20260812 11:10:48.752928 17003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:10:48.755455 17003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:10:48.756779 17003 hybrid_clock.cc:648] HybridClock initialized: now 1786533048756650 us; error 100 us; skew 500 ppm
I20260812 11:10:48.757506 17003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:10:48.762056 17003 webserver.cc:460] Webserver started at http://127.16.151.190:43811/ using document root <none> and password file <none>
I20260812 11:10:48.763309 17003 fs_manager.cc:359] Metadata directory not provided
I20260812 11:10:48.763530 17003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:10:48.764035 17003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:10:48.768132 17003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/instance:
uuid: "ad68f6c35c69479391bebe4dcd9e223c"
format_stamp: "Formatted at 2026-08-12 11:10:48 on dist-test-slave-bw88"
I20260812 11:10:48.769021 17003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/wal/instance:
uuid: "ad68f6c35c69479391bebe4dcd9e223c"
format_stamp: "Formatted at 2026-08-12 11:10:48 on dist-test-slave-bw88"
I20260812 11:10:48.778057 17003 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260812 11:10:48.784828 17027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:10:48.786859 17003 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 11:10:48.787199 17003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data,/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/wal
uuid: "ad68f6c35c69479391bebe4dcd9e223c"
format_stamp: "Formatted at 2026-08-12 11:10:48 on dist-test-slave-bw88"
I20260812 11:10:48.787493 17003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:10:48.809451 17003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:10:48.810844 17003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:10:48.811432 17003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:10:48.850845 17003 rpc_server.cc:307] RPC server started. Bound to: 127.16.151.190:43065
I20260812 11:10:48.850920 17108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.151.190:43065 every 8 connection(s)
I20260812 11:10:48.853422 17003 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb
I20260812 11:10:48.855852 16990 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknusDu0/build/asan/bin/kudu as pid 17003
I20260812 11:10:48.856290 16990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/wal/instance
I20260812 11:10:48.862304 17109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:10:48.884238 17109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad68f6c35c69479391bebe4dcd9e223c: Bootstrap starting.
I20260812 11:10:48.890599 17109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad68f6c35c69479391bebe4dcd9e223c: Neither blocks nor log segments found. Creating new log.
I20260812 11:10:48.892858 17109 log.cc:824] T 00000000000000000000000000000000 P ad68f6c35c69479391bebe4dcd9e223c: Log is configured to *not* fsync() on all Append() calls
I20260812 11:10:48.898262 17109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad68f6c35c69479391bebe4dcd9e223c: No bootstrap required, opened a new log
I20260812 11:10:48.910327 17109 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad68f6c35c69479391bebe4dcd9e223c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad68f6c35c69479391bebe4dcd9e223c" member_type: VOTER last_known_addr { host: "127.16.151.190" port: 43065 } }
I20260812 11:10:48.911012 17109 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad68f6c35c69479391bebe4dcd9e223c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:10:48.911401 17109 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad68f6c35c69479391bebe4dcd9e223c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad68f6c35c69479391bebe4dcd9e223c, State: Initialized, Role: FOLLOWER
I20260812 11:10:48.912842 17109 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad68f6c35c69479391bebe4dcd9e223c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad68f6c35c69479391bebe4dcd9e223c" member_type: VOTER last_known_addr { host: "127.16.151.190" port: 43065 } }
I20260812 11:10:48.913393 17109 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad68f6c35c69479391bebe4dcd9e223c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:10:48.913798 17109 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad68f6c35c69479391bebe4dcd9e223c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:10:48.914085 17109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad68f6c35c69479391bebe4dcd9e223c [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:10:48.916826 17109 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad68f6c35c69479391bebe4dcd9e223c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad68f6c35c69479391bebe4dcd9e223c" member_type: VOTER last_known_addr { host: "127.16.151.190" port: 43065 } }
I20260812 11:10:48.918177 17109 leader_election.cc:302] T 00000000000000000000000000000000 P ad68f6c35c69479391bebe4dcd9e223c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad68f6c35c69479391bebe4dcd9e223c; no voters:
I20260812 11:10:48.919581 17109 leader_election.cc:288] T 00000000000000000000000000000000 P ad68f6c35c69479391bebe4dcd9e223c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:10:48.919781 17115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad68f6c35c69479391bebe4dcd9e223c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:10:48.921566 17115 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad68f6c35c69479391bebe4dcd9e223c [term 1 LEADER]: Becoming Leader. State: Replica: ad68f6c35c69479391bebe4dcd9e223c, State: Running, Role: LEADER
I20260812 11:10:48.922544 17115 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad68f6c35c69479391bebe4dcd9e223c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad68f6c35c69479391bebe4dcd9e223c" member_type: VOTER last_known_addr { host: "127.16.151.190" port: 43065 } }
I20260812 11:10:48.923702 17109 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad68f6c35c69479391bebe4dcd9e223c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:10:48.931299 17116 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad68f6c35c69479391bebe4dcd9e223c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad68f6c35c69479391bebe4dcd9e223c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad68f6c35c69479391bebe4dcd9e223c" member_type: VOTER last_known_addr { host: "127.16.151.190" port: 43065 } } }
I20260812 11:10:48.931172 17119 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad68f6c35c69479391bebe4dcd9e223c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad68f6c35c69479391bebe4dcd9e223c. Latest consensus state: current_term: 1 leader_uuid: "ad68f6c35c69479391bebe4dcd9e223c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad68f6c35c69479391bebe4dcd9e223c" member_type: VOTER last_known_addr { host: "127.16.151.190" port: 43065 } } }
I20260812 11:10:48.932355 17116 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad68f6c35c69479391bebe4dcd9e223c [sys.catalog]: This master's current role is: LEADER
I20260812 11:10:48.932514 17119 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad68f6c35c69479391bebe4dcd9e223c [sys.catalog]: This master's current role is: LEADER
I20260812 11:10:48.940852 17132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:10:48.951160 17132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:10:48.968604 17132 catalog_manager.cc:1347] Generated new cluster ID: 43824d0ad82546d6903b86b6b0663727
I20260812 11:10:48.968811 17132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:10:48.988787 17132 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:10:48.990785 17132 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:10:49.019635 17132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad68f6c35c69479391bebe4dcd9e223c: Generated new TSK 0
I20260812 11:10:49.020884 17132 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:10:49.030328 16990 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknusDu0/build/asan/bin/kudu
/tmp/dist-test-tasknusDu0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.151.129:0
--local_ip_for_outbound_sockets=127.16.151.129
--webserver_interface=127.16.151.129
--webserver_port=0
--tserver_master_addrs=127.16.151.190:43065
--builtin_ntp_servers=127.16.151.148:37557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
W20260812 11:10:49.583424 17147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:10:49.583778 17147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:10:49.583962 17147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:10:49.591351 17147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:10:49.591540 17147 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260812 11:10:49.591639 17147 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260812 11:10:49.591800 17147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.151.129
I20260812 11:10:49.601675 17147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.151.148:37557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.151.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.151.129
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.16.151.190:43065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.151.129
--log_dir=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:10:49.603140 17147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:10:49.605083 17147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:10:49.621016 17159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:10:49.621116 17155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:10:49.621112 17156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:10:49.622196 17147 server_base.cc:1034] running on GCE node
I20260812 11:10:49.622933 17147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:10:49.624037 17147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:10:49.625262 17147 hybrid_clock.cc:648] HybridClock initialized: now 1786533049625192 us; error 38 us; skew 500 ppm
I20260812 11:10:49.625692 17147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:10:49.633956 17147 webserver.cc:460] Webserver started at http://127.16.151.129:35159/ using document root <none> and password file <none>
I20260812 11:10:49.634898 17147 fs_manager.cc:359] Metadata directory not provided
I20260812 11:10:49.635094 17147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:10:49.635532 17147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:10:49.639133 17147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/instance:
uuid: "b7094e9660dc47ceb5e9f70e089ef166"
format_stamp: "Formatted at 2026-08-12 11:10:49 on dist-test-slave-bw88"
I20260812 11:10:49.640175 17147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/wal/instance:
uuid: "b7094e9660dc47ceb5e9f70e089ef166"
format_stamp: "Formatted at 2026-08-12 11:10:49 on dist-test-slave-bw88"
I20260812 11:10:49.647230 17147 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 11:10:49.651856 17167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:10:49.653601 17147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:10:49.653826 17147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/wal
uuid: "b7094e9660dc47ceb5e9f70e089ef166"
format_stamp: "Formatted at 2026-08-12 11:10:49 on dist-test-slave-bw88"
I20260812 11:10:49.654071 17147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:10:49.679569 17147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:10:49.680529 17147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:10:49.680881 17147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:10:49.682164 17147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:10:49.684244 17147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:10:49.684327 17147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:10:49.684544 17147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:10:49.684609 17147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:10:49.731132 17147 rpc_server.cc:307] RPC server started. Bound to: 127.16.151.129:42789
I20260812 11:10:49.731259 17342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.151.129:42789 every 8 connection(s)
I20260812 11:10:49.732694 17147 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb
I20260812 11:10:49.741554 16990 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknusDu0/build/asan/bin/kudu as pid 17147
I20260812 11:10:49.741777 16990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/wal/instance
I20260812 11:10:49.746273 16990 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknusDu0/build/asan/bin/kudu
/tmp/dist-test-tasknusDu0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.151.130:0
--local_ip_for_outbound_sockets=127.16.151.130
--webserver_interface=127.16.151.130
--webserver_port=0
--tserver_master_addrs=127.16.151.190:43065
--builtin_ntp_servers=127.16.151.148:37557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
I20260812 11:10:49.752975 17343 heartbeater.cc:347] Connected to a master server at 127.16.151.190:43065
I20260812 11:10:49.753584 17343 heartbeater.cc:464] Registering TS with master...
I20260812 11:10:49.754690 17343 heartbeater.cc:511] Master 127.16.151.190:43065 requested a full tablet report, sending...
I20260812 11:10:49.758852 17053 ts_manager.cc:194] Registered new tserver with Master: b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789)
I20260812 11:10:49.761426 17053 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.151.129:59615
W20260812 11:10:50.223174 17347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:10:50.223493 17347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:10:50.223675 17347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:10:50.232887 17347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:10:50.233040 17347 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260812 11:10:50.233202 17347 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260812 11:10:50.233496 17347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.151.130
I20260812 11:10:50.244714 17347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.151.148:37557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.151.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.151.130
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.16.151.190:43065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.151.130
--log_dir=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:10:50.246253 17347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:10:50.248303 17347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:10:50.266232 17353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:10:50.266291 17359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:10:50.266374 17355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:10:50.267755 17347 server_base.cc:1034] running on GCE node
I20260812 11:10:50.268474 17347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:10:50.269585 17347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:10:50.270968 17347 hybrid_clock.cc:648] HybridClock initialized: now 1786533050270832 us; error 63 us; skew 500 ppm
I20260812 11:10:50.271497 17347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:10:50.275249 17347 webserver.cc:460] Webserver started at http://127.16.151.130:38721/ using document root <none> and password file <none>
I20260812 11:10:50.276029 17347 fs_manager.cc:359] Metadata directory not provided
I20260812 11:10:50.276181 17347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:10:50.276487 17347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:10:50.279458 17347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/instance:
uuid: "3884e2014802405c8f38db8f6dcabc25"
format_stamp: "Formatted at 2026-08-12 11:10:50 on dist-test-slave-bw88"
I20260812 11:10:50.280289 17347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/wal/instance:
uuid: "3884e2014802405c8f38db8f6dcabc25"
format_stamp: "Formatted at 2026-08-12 11:10:50 on dist-test-slave-bw88"
I20260812 11:10:50.286188 17347 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20260812 11:10:50.289815 17366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:10:50.291234 17347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 11:10:50.291414 17347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/wal
uuid: "3884e2014802405c8f38db8f6dcabc25"
format_stamp: "Formatted at 2026-08-12 11:10:50 on dist-test-slave-bw88"
I20260812 11:10:50.291616 17347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:10:50.323240 17347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:10:50.324257 17347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:10:50.324587 17347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:10:50.325964 17347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:10:50.327854 17347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:10:50.328004 17347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:10:50.328266 17347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:10:50.328373 17347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:10:50.389019 17347 rpc_server.cc:307] RPC server started. Bound to: 127.16.151.130:37959
I20260812 11:10:50.389122 17540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.151.130:37959 every 8 connection(s)
I20260812 11:10:50.390614 17347 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb
I20260812 11:10:50.393534 16990 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknusDu0/build/asan/bin/kudu as pid 17347
I20260812 11:10:50.393797 16990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/wal/instance
I20260812 11:10:50.398974 16990 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknusDu0/build/asan/bin/kudu
/tmp/dist-test-tasknusDu0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.151.131:0
--local_ip_for_outbound_sockets=127.16.151.131
--webserver_interface=127.16.151.131
--webserver_port=0
--tserver_master_addrs=127.16.151.190:43065
--builtin_ntp_servers=127.16.151.148:37557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
I20260812 11:10:50.411320 17541 heartbeater.cc:347] Connected to a master server at 127.16.151.190:43065
I20260812 11:10:50.411675 17541 heartbeater.cc:464] Registering TS with master...
I20260812 11:10:50.412602 17541 heartbeater.cc:511] Master 127.16.151.190:43065 requested a full tablet report, sending...
I20260812 11:10:50.414412 17053 ts_manager.cc:194] Registered new tserver with Master: 3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959)
I20260812 11:10:50.415424 17053 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.151.130:51489
I20260812 11:10:50.765626 17343 heartbeater.cc:503] Master 127.16.151.190:43065 was elected leader, sending a full tablet report...
W20260812 11:10:50.870589 17546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:10:50.870971 17546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:10:50.871210 17546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:10:50.879611 17546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:10:50.879822 17546 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260812 11:10:50.879907 17546 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260812 11:10:50.880069 17546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.151.131
I20260812 11:10:50.890923 17546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.151.148:37557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.151.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.151.131
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.16.151.190:43065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.151.131
--log_dir=/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:10:50.892776 17546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:10:50.894831 17546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:10:50.911820 17553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:10:50.911746 17555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:10:50.912631 17552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:10:50.913378 17546 server_base.cc:1034] running on GCE node
I20260812 11:10:50.914187 17546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:10:50.915246 17546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:10:50.916563 17546 hybrid_clock.cc:648] HybridClock initialized: now 1786533050916400 us; error 84 us; skew 500 ppm
I20260812 11:10:50.916896 17546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:10:50.920169 17546 webserver.cc:460] Webserver started at http://127.16.151.131:40329/ using document root <none> and password file <none>
I20260812 11:10:50.920920 17546 fs_manager.cc:359] Metadata directory not provided
I20260812 11:10:50.921130 17546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:10:50.921655 17546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:10:50.924551 17546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-2/data/instance:
uuid: "5438c5a3070f40d99bb7955dc412fbcc"
format_stamp: "Formatted at 2026-08-12 11:10:50 on dist-test-slave-bw88"
I20260812 11:10:50.925557 17546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-2/wal/instance:
uuid: "5438c5a3070f40d99bb7955dc412fbcc"
format_stamp: "Formatted at 2026-08-12 11:10:50 on dist-test-slave-bw88"
I20260812 11:10:50.932005 17546 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260812 11:10:50.936254 17565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:10:50.937769 17546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 11:10:50.937958 17546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-2/wal
uuid: "5438c5a3070f40d99bb7955dc412fbcc"
format_stamp: "Formatted at 2026-08-12 11:10:50 on dist-test-slave-bw88"
I20260812 11:10:50.938184 17546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:10:50.958442 17546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:10:50.959578 17546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:10:50.960146 17546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:10:50.961824 17546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:10:50.963764 17546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:10:50.963891 17546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:10:50.964067 17546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:10:50.964154 17546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:10:51.022694 17546 rpc_server.cc:307] RPC server started. Bound to: 127.16.151.131:38427
I20260812 11:10:51.022758 17736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.151.131:38427 every 8 connection(s)
I20260812 11:10:51.024205 17546 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-2/data/info.pb
I20260812 11:10:51.027030 16990 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknusDu0/build/asan/bin/kudu as pid 17546
I20260812 11:10:51.027251 16990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-2/wal/instance
I20260812 11:10:51.041594 17737 heartbeater.cc:347] Connected to a master server at 127.16.151.190:43065
I20260812 11:10:51.041965 17737 heartbeater.cc:464] Registering TS with master...
I20260812 11:10:51.043017 17737 heartbeater.cc:511] Master 127.16.151.190:43065 requested a full tablet report, sending...
I20260812 11:10:51.044749 17053 ts_manager.cc:194] Registered new tserver with Master: 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427)
I20260812 11:10:51.045805 17053 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.151.131:45203
I20260812 11:10:51.046393 16990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:10:51.075407 17053 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37570:
name: "all-types-table"
schema {
columns {
name: "key"
type: INT16
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int8_val"
type: INT8
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int16_val"
type: INT16
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int32_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int64_val"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "timestamp_val"
type: UNIXTIME_MICROS
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "date_val"
type: DATE
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "varchar_val"
type: VARCHAR
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
length: 65535
}
immutable: false
}
columns {
name: "bool_val"
type: BOOL
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "float_val"
type: FLOAT
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "double_val"
type: DOUBLE
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "binary_val"
type: BINARY
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "decimal32_val"
type: DECIMAL32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 9
scale: 0
}
immutable: false
}
columns {
name: "decimal64_val"
type: DECIMAL64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 18
scale: 0
}
immutable: false
}
columns {
name: "decimal128_val"
type: DECIMAL128
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 38
scale: 0
}
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\000\000\252*\004\001\000\000\000TU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:10:51.088749 17053 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table all-types-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:10:51.134720 17643 tablet_service.cc:1467] Processing CreateTablet for tablet 6dda8ac5b4254f1296f26326220247e2 (DEFAULT_TABLE table=all-types-table [id=b6d6fbd886264139bf8ec2888134074c]), partition=RANGE (key) PARTITION VALUES < 10922
I20260812 11:10:51.136252 17642 tablet_service.cc:1467] Processing CreateTablet for tablet a99dd1ffecd04818b07c06d01f830f1a (DEFAULT_TABLE table=all-types-table [id=b6d6fbd886264139bf8ec2888134074c]), partition=RANGE (key) PARTITION 10922 <= VALUES < 21844
I20260812 11:10:51.137456 17641 tablet_service.cc:1467] Processing CreateTablet for tablet 88f0bbdf7cac496198e6b83299d50850 (DEFAULT_TABLE table=all-types-table [id=b6d6fbd886264139bf8ec2888134074c]), partition=RANGE (key) PARTITION 21844 <= VALUES
I20260812 11:10:51.137854 17643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dda8ac5b4254f1296f26326220247e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:10:51.138326 17641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f0bbdf7cac496198e6b83299d50850. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:10:51.137071 17246 tablet_service.cc:1467] Processing CreateTablet for tablet 6dda8ac5b4254f1296f26326220247e2 (DEFAULT_TABLE table=all-types-table [id=b6d6fbd886264139bf8ec2888134074c]), partition=RANGE (key) PARTITION VALUES < 10922
I20260812 11:10:51.137012 17241 tablet_service.cc:1467] Processing CreateTablet for tablet 88f0bbdf7cac496198e6b83299d50850 (DEFAULT_TABLE table=all-types-table [id=b6d6fbd886264139bf8ec2888134074c]), partition=RANGE (key) PARTITION 21844 <= VALUES
I20260812 11:10:51.138731 17442 tablet_service.cc:1467] Processing CreateTablet for tablet a99dd1ffecd04818b07c06d01f830f1a (DEFAULT_TABLE table=all-types-table [id=b6d6fbd886264139bf8ec2888134074c]), partition=RANGE (key) PARTITION 10922 <= VALUES < 21844
I20260812 11:10:51.140514 17246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dda8ac5b4254f1296f26326220247e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:10:51.141045 17241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f0bbdf7cac496198e6b83299d50850. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:10:51.141749 17642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a99dd1ffecd04818b07c06d01f830f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:10:51.142095 17245 tablet_service.cc:1467] Processing CreateTablet for tablet a99dd1ffecd04818b07c06d01f830f1a (DEFAULT_TABLE table=all-types-table [id=b6d6fbd886264139bf8ec2888134074c]), partition=RANGE (key) PARTITION 10922 <= VALUES < 21844
I20260812 11:10:51.142725 17245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a99dd1ffecd04818b07c06d01f830f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:10:51.143177 17441 tablet_service.cc:1467] Processing CreateTablet for tablet 88f0bbdf7cac496198e6b83299d50850 (DEFAULT_TABLE table=all-types-table [id=b6d6fbd886264139bf8ec2888134074c]), partition=RANGE (key) PARTITION 21844 <= VALUES
I20260812 11:10:51.143257 17443 tablet_service.cc:1467] Processing CreateTablet for tablet 6dda8ac5b4254f1296f26326220247e2 (DEFAULT_TABLE table=all-types-table [id=b6d6fbd886264139bf8ec2888134074c]), partition=RANGE (key) PARTITION VALUES < 10922
I20260812 11:10:51.144304 17442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a99dd1ffecd04818b07c06d01f830f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:10:51.144685 17441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f0bbdf7cac496198e6b83299d50850. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:10:51.146255 17443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dda8ac5b4254f1296f26326220247e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:10:51.162324 17776 tablet_bootstrap.cc:492] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc: Bootstrap starting.
I20260812 11:10:51.162518 17777 tablet_bootstrap.cc:492] T 6dda8ac5b4254f1296f26326220247e2 P 3884e2014802405c8f38db8f6dcabc25: Bootstrap starting.
I20260812 11:10:51.167127 17777 tablet_bootstrap.cc:654] T 6dda8ac5b4254f1296f26326220247e2 P 3884e2014802405c8f38db8f6dcabc25: Neither blocks nor log segments found. Creating new log.
I20260812 11:10:51.167372 17776 tablet_bootstrap.cc:654] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc: Neither blocks nor log segments found. Creating new log.
I20260812 11:10:51.168736 17777 log.cc:824] T 6dda8ac5b4254f1296f26326220247e2 P 3884e2014802405c8f38db8f6dcabc25: Log is configured to *not* fsync() on all Append() calls
I20260812 11:10:51.169106 17776 log.cc:824] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc: Log is configured to *not* fsync() on all Append() calls
I20260812 11:10:51.172735 17777 tablet_bootstrap.cc:492] T 6dda8ac5b4254f1296f26326220247e2 P 3884e2014802405c8f38db8f6dcabc25: No bootstrap required, opened a new log
I20260812 11:10:51.173170 17777 ts_tablet_manager.cc:1397] T 6dda8ac5b4254f1296f26326220247e2 P 3884e2014802405c8f38db8f6dcabc25: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260812 11:10:51.187850 17785 tablet_bootstrap.cc:492] T 88f0bbdf7cac496198e6b83299d50850 P b7094e9660dc47ceb5e9f70e089ef166: Bootstrap starting.
I20260812 11:10:51.189028 17776 tablet_bootstrap.cc:492] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc: No bootstrap required, opened a new log
I20260812 11:10:51.189424 17776 ts_tablet_manager.cc:1397] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.014s
I20260812 11:10:51.189529 17777 raft_consensus.cc:348] T 6dda8ac5b4254f1296f26326220247e2 P 3884e2014802405c8f38db8f6dcabc25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } }
I20260812 11:10:51.190264 17777 raft_consensus.cc:374] T 6dda8ac5b4254f1296f26326220247e2 P 3884e2014802405c8f38db8f6dcabc25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:10:51.190462 17777 raft_consensus.cc:729] T 6dda8ac5b4254f1296f26326220247e2 P 3884e2014802405c8f38db8f6dcabc25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3884e2014802405c8f38db8f6dcabc25, State: Initialized, Role: FOLLOWER
I20260812 11:10:51.191308 17777 consensus_queue.cc:260] T 6dda8ac5b4254f1296f26326220247e2 P 3884e2014802405c8f38db8f6dcabc25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } }
I20260812 11:10:51.192998 17785 tablet_bootstrap.cc:654] T 88f0bbdf7cac496198e6b83299d50850 P b7094e9660dc47ceb5e9f70e089ef166: Neither blocks nor log segments found. Creating new log.
I20260812 11:10:51.192971 17541 heartbeater.cc:503] Master 127.16.151.190:43065 was elected leader, sending a full tablet report...
I20260812 11:10:51.193910 17777 ts_tablet_manager.cc:1428] T 6dda8ac5b4254f1296f26326220247e2 P 3884e2014802405c8f38db8f6dcabc25: Time spent starting tablet: real 0.020s user 0.018s sys 0.000s
I20260812 11:10:51.195035 17785 log.cc:824] T 88f0bbdf7cac496198e6b83299d50850 P b7094e9660dc47ceb5e9f70e089ef166: Log is configured to *not* fsync() on all Append() calls
I20260812 11:10:51.196264 17777 tablet_bootstrap.cc:492] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25: Bootstrap starting.
I20260812 11:10:51.199082 17777 tablet_bootstrap.cc:654] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25: Neither blocks nor log segments found. Creating new log.
I20260812 11:10:51.202337 17776 raft_consensus.cc:348] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } }
I20260812 11:10:51.203012 17776 raft_consensus.cc:374] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:10:51.203176 17776 raft_consensus.cc:729] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5438c5a3070f40d99bb7955dc412fbcc, State: Initialized, Role: FOLLOWER
I20260812 11:10:51.203938 17776 consensus_queue.cc:260] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } }
I20260812 11:10:51.204640 17777 tablet_bootstrap.cc:492] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25: No bootstrap required, opened a new log
I20260812 11:10:51.204805 17777 ts_tablet_manager.cc:1397] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260812 11:10:51.205310 17737 heartbeater.cc:503] Master 127.16.151.190:43065 was elected leader, sending a full tablet report...
I20260812 11:10:51.205905 17777 raft_consensus.cc:348] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } }
I20260812 11:10:51.206142 17777 raft_consensus.cc:374] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:10:51.206187 17776 ts_tablet_manager.cc:1428] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc: Time spent starting tablet: real 0.017s user 0.006s sys 0.009s
I20260812 11:10:51.206252 17777 raft_consensus.cc:729] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3884e2014802405c8f38db8f6dcabc25, State: Initialized, Role: FOLLOWER
I20260812 11:10:51.206614 17776 tablet_bootstrap.cc:492] T 6dda8ac5b4254f1296f26326220247e2 P 5438c5a3070f40d99bb7955dc412fbcc: Bootstrap starting.
I20260812 11:10:51.206552 17777 consensus_queue.cc:260] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } }
I20260812 11:10:51.207357 17777 ts_tablet_manager.cc:1428] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 11:10:51.207686 17777 tablet_bootstrap.cc:492] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25: Bootstrap starting.
I20260812 11:10:51.207733 17785 tablet_bootstrap.cc:492] T 88f0bbdf7cac496198e6b83299d50850 P b7094e9660dc47ceb5e9f70e089ef166: No bootstrap required, opened a new log
I20260812 11:10:51.208137 17785 ts_tablet_manager.cc:1397] T 88f0bbdf7cac496198e6b83299d50850 P b7094e9660dc47ceb5e9f70e089ef166: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.004s
I20260812 11:10:51.208532 17776 tablet_bootstrap.cc:654] T 6dda8ac5b4254f1296f26326220247e2 P 5438c5a3070f40d99bb7955dc412fbcc: Neither blocks nor log segments found. Creating new log.
I20260812 11:10:51.209630 17777 tablet_bootstrap.cc:654] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25: Neither blocks nor log segments found. Creating new log.
I20260812 11:10:51.217406 17776 tablet_bootstrap.cc:492] T 6dda8ac5b4254f1296f26326220247e2 P 5438c5a3070f40d99bb7955dc412fbcc: No bootstrap required, opened a new log
I20260812 11:10:51.217722 17776 ts_tablet_manager.cc:1397] T 6dda8ac5b4254f1296f26326220247e2 P 5438c5a3070f40d99bb7955dc412fbcc: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.005s
I20260812 11:10:51.218894 17776 raft_consensus.cc:348] T 6dda8ac5b4254f1296f26326220247e2 P 5438c5a3070f40d99bb7955dc412fbcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } }
I20260812 11:10:51.219139 17776 raft_consensus.cc:374] T 6dda8ac5b4254f1296f26326220247e2 P 5438c5a3070f40d99bb7955dc412fbcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:10:51.219246 17776 raft_consensus.cc:729] T 6dda8ac5b4254f1296f26326220247e2 P 5438c5a3070f40d99bb7955dc412fbcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5438c5a3070f40d99bb7955dc412fbcc, State: Initialized, Role: FOLLOWER
I20260812 11:10:51.219533 17776 consensus_queue.cc:260] T 6dda8ac5b4254f1296f26326220247e2 P 5438c5a3070f40d99bb7955dc412fbcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } }
I20260812 11:10:51.219877 17777 tablet_bootstrap.cc:492] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25: No bootstrap required, opened a new log
I20260812 11:10:51.220088 17777 ts_tablet_manager.cc:1397] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20260812 11:10:51.220157 17776 ts_tablet_manager.cc:1428] T 6dda8ac5b4254f1296f26326220247e2 P 5438c5a3070f40d99bb7955dc412fbcc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 11:10:51.220449 17776 tablet_bootstrap.cc:492] T 88f0bbdf7cac496198e6b83299d50850 P 5438c5a3070f40d99bb7955dc412fbcc: Bootstrap starting.
I20260812 11:10:51.221073 17777 raft_consensus.cc:348] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } }
I20260812 11:10:51.221481 17777 raft_consensus.cc:374] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:10:51.221576 17777 raft_consensus.cc:729] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3884e2014802405c8f38db8f6dcabc25, State: Initialized, Role: FOLLOWER
I20260812 11:10:51.221956 17777 consensus_queue.cc:260] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } }
I20260812 11:10:51.222767 17776 tablet_bootstrap.cc:654] T 88f0bbdf7cac496198e6b83299d50850 P 5438c5a3070f40d99bb7955dc412fbcc: Neither blocks nor log segments found. Creating new log.
I20260812 11:10:51.222787 17785 raft_consensus.cc:348] T 88f0bbdf7cac496198e6b83299d50850 P b7094e9660dc47ceb5e9f70e089ef166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } }
I20260812 11:10:51.223367 17785 raft_consensus.cc:374] T 88f0bbdf7cac496198e6b83299d50850 P b7094e9660dc47ceb5e9f70e089ef166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:10:51.223480 17785 raft_consensus.cc:729] T 88f0bbdf7cac496198e6b83299d50850 P b7094e9660dc47ceb5e9f70e089ef166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7094e9660dc47ceb5e9f70e089ef166, State: Initialized, Role: FOLLOWER
I20260812 11:10:51.224462 17785 consensus_queue.cc:260] T 88f0bbdf7cac496198e6b83299d50850 P b7094e9660dc47ceb5e9f70e089ef166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } }
I20260812 11:10:51.225044 17777 ts_tablet_manager.cc:1428] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260812 11:10:51.227703 17785 ts_tablet_manager.cc:1428] T 88f0bbdf7cac496198e6b83299d50850 P b7094e9660dc47ceb5e9f70e089ef166: Time spent starting tablet: real 0.019s user 0.018s sys 0.000s
I20260812 11:10:51.228765 17785 tablet_bootstrap.cc:492] T a99dd1ffecd04818b07c06d01f830f1a P b7094e9660dc47ceb5e9f70e089ef166: Bootstrap starting.
I20260812 11:10:51.231383 17776 tablet_bootstrap.cc:492] T 88f0bbdf7cac496198e6b83299d50850 P 5438c5a3070f40d99bb7955dc412fbcc: No bootstrap required, opened a new log
I20260812 11:10:51.231434 17785 tablet_bootstrap.cc:654] T a99dd1ffecd04818b07c06d01f830f1a P b7094e9660dc47ceb5e9f70e089ef166: Neither blocks nor log segments found. Creating new log.
I20260812 11:10:51.231626 17776 ts_tablet_manager.cc:1397] T 88f0bbdf7cac496198e6b83299d50850 P 5438c5a3070f40d99bb7955dc412fbcc: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260812 11:10:51.232864 17776 raft_consensus.cc:348] T 88f0bbdf7cac496198e6b83299d50850 P 5438c5a3070f40d99bb7955dc412fbcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } }
I20260812 11:10:51.233172 17776 raft_consensus.cc:374] T 88f0bbdf7cac496198e6b83299d50850 P 5438c5a3070f40d99bb7955dc412fbcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:10:51.233281 17776 raft_consensus.cc:729] T 88f0bbdf7cac496198e6b83299d50850 P 5438c5a3070f40d99bb7955dc412fbcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5438c5a3070f40d99bb7955dc412fbcc, State: Initialized, Role: FOLLOWER
I20260812 11:10:51.233695 17776 consensus_queue.cc:260] T 88f0bbdf7cac496198e6b83299d50850 P 5438c5a3070f40d99bb7955dc412fbcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } }
I20260812 11:10:51.234380 17776 ts_tablet_manager.cc:1428] T 88f0bbdf7cac496198e6b83299d50850 P 5438c5a3070f40d99bb7955dc412fbcc: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260812 11:10:51.234848 17785 tablet_bootstrap.cc:492] T a99dd1ffecd04818b07c06d01f830f1a P b7094e9660dc47ceb5e9f70e089ef166: No bootstrap required, opened a new log
I20260812 11:10:51.235086 17785 ts_tablet_manager.cc:1397] T a99dd1ffecd04818b07c06d01f830f1a P b7094e9660dc47ceb5e9f70e089ef166: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
W20260812 11:10:51.236356 17344 tablet.cc:2369] T 88f0bbdf7cac496198e6b83299d50850 P b7094e9660dc47ceb5e9f70e089ef166: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:10:51.236295 17785 raft_consensus.cc:348] T a99dd1ffecd04818b07c06d01f830f1a P b7094e9660dc47ceb5e9f70e089ef166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } }
I20260812 11:10:51.236701 17785 raft_consensus.cc:374] T a99dd1ffecd04818b07c06d01f830f1a P b7094e9660dc47ceb5e9f70e089ef166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:10:51.236882 17785 raft_consensus.cc:729] T a99dd1ffecd04818b07c06d01f830f1a P b7094e9660dc47ceb5e9f70e089ef166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7094e9660dc47ceb5e9f70e089ef166, State: Initialized, Role: FOLLOWER
I20260812 11:10:51.237229 17785 consensus_queue.cc:260] T a99dd1ffecd04818b07c06d01f830f1a P b7094e9660dc47ceb5e9f70e089ef166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } }
I20260812 11:10:51.237946 17785 ts_tablet_manager.cc:1428] T a99dd1ffecd04818b07c06d01f830f1a P b7094e9660dc47ceb5e9f70e089ef166: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260812 11:10:51.238325 17785 tablet_bootstrap.cc:492] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166: Bootstrap starting.
I20260812 11:10:51.240662 17785 tablet_bootstrap.cc:654] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166: Neither blocks nor log segments found. Creating new log.
I20260812 11:10:51.243237 17785 tablet_bootstrap.cc:492] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166: No bootstrap required, opened a new log
I20260812 11:10:51.243484 17785 ts_tablet_manager.cc:1397] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260812 11:10:51.244158 17785 raft_consensus.cc:348] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } }
I20260812 11:10:51.244370 17785 raft_consensus.cc:374] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:10:51.244452 17785 raft_consensus.cc:729] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7094e9660dc47ceb5e9f70e089ef166, State: Initialized, Role: FOLLOWER
I20260812 11:10:51.244727 17785 consensus_queue.cc:260] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } }
I20260812 11:10:51.245177 17785 ts_tablet_manager.cc:1428] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20260812 11:10:51.276846 17738 tablet.cc:2369] T 6dda8ac5b4254f1296f26326220247e2 P 5438c5a3070f40d99bb7955dc412fbcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:10:51.315335 17787 raft_consensus.cc:482] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:10:51.316051 17787 raft_consensus.cc:504] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } }
I20260812 11:10:51.318522 17787 leader_election.cc:288] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789), 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427)
I20260812 11:10:51.330018 17672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f0bbdf7cac496198e6b83299d50850" candidate_uuid: "3884e2014802405c8f38db8f6dcabc25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5438c5a3070f40d99bb7955dc412fbcc" is_pre_election: true
I20260812 11:10:51.330994 17672 raft_consensus.cc:2454] T 88f0bbdf7cac496198e6b83299d50850 P 5438c5a3070f40d99bb7955dc412fbcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3884e2014802405c8f38db8f6dcabc25 in term 0.
I20260812 11:10:51.330876 17270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f0bbdf7cac496198e6b83299d50850" candidate_uuid: "3884e2014802405c8f38db8f6dcabc25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7094e9660dc47ceb5e9f70e089ef166" is_pre_election: true
I20260812 11:10:51.331501 17270 raft_consensus.cc:2454] T 88f0bbdf7cac496198e6b83299d50850 P b7094e9660dc47ceb5e9f70e089ef166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3884e2014802405c8f38db8f6dcabc25 in term 0.
I20260812 11:10:51.332041 17371 leader_election.cc:302] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3884e2014802405c8f38db8f6dcabc25, 5438c5a3070f40d99bb7955dc412fbcc; no voters:
I20260812 11:10:51.332665 17787 raft_consensus.cc:2781] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:10:51.332930 17787 raft_consensus.cc:482] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:10:51.333107 17787 raft_consensus.cc:3037] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:10:51.336190 17787 raft_consensus.cc:504] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } }
I20260812 11:10:51.336961 17787 leader_election.cc:288] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 [CANDIDATE]: Term 1 election: Requested vote from peers b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789), 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427)
I20260812 11:10:51.337553 17672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f0bbdf7cac496198e6b83299d50850" candidate_uuid: "3884e2014802405c8f38db8f6dcabc25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5438c5a3070f40d99bb7955dc412fbcc"
I20260812 11:10:51.337530 17270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f0bbdf7cac496198e6b83299d50850" candidate_uuid: "3884e2014802405c8f38db8f6dcabc25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7094e9660dc47ceb5e9f70e089ef166"
I20260812 11:10:51.337864 17672 raft_consensus.cc:3037] T 88f0bbdf7cac496198e6b83299d50850 P 5438c5a3070f40d99bb7955dc412fbcc [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:10:51.338039 17270 raft_consensus.cc:3037] T 88f0bbdf7cac496198e6b83299d50850 P b7094e9660dc47ceb5e9f70e089ef166 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:10:51.340510 17672 raft_consensus.cc:2454] T 88f0bbdf7cac496198e6b83299d50850 P 5438c5a3070f40d99bb7955dc412fbcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3884e2014802405c8f38db8f6dcabc25 in term 1.
I20260812 11:10:51.340595 17270 raft_consensus.cc:2454] T 88f0bbdf7cac496198e6b83299d50850 P b7094e9660dc47ceb5e9f70e089ef166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3884e2014802405c8f38db8f6dcabc25 in term 1.
I20260812 11:10:51.340989 17371 leader_election.cc:302] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3884e2014802405c8f38db8f6dcabc25, 5438c5a3070f40d99bb7955dc412fbcc; no voters:
I20260812 11:10:51.341462 17787 raft_consensus.cc:2781] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:10:51.342156 17787 raft_consensus.cc:686] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 [term 1 LEADER]: Becoming Leader. State: Replica: 3884e2014802405c8f38db8f6dcabc25, State: Running, Role: LEADER
I20260812 11:10:51.342711 17787 consensus_queue.cc:237] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } }
I20260812 11:10:51.348624 17053 catalog_manager.cc:5495] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3884e2014802405c8f38db8f6dcabc25 (127.16.151.130). New cstate: current_term: 1 leader_uuid: "3884e2014802405c8f38db8f6dcabc25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } health_report { overall_health: UNKNOWN } } }
I20260812 11:10:51.350642 17791 raft_consensus.cc:482] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:10:51.350951 17791 raft_consensus.cc:504] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } }
I20260812 11:10:51.353219 17791 leader_election.cc:288] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789), 3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959)
I20260812 11:10:51.363090 17270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a99dd1ffecd04818b07c06d01f830f1a" candidate_uuid: "5438c5a3070f40d99bb7955dc412fbcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7094e9660dc47ceb5e9f70e089ef166" is_pre_election: true
I20260812 11:10:51.363495 17270 raft_consensus.cc:2454] T a99dd1ffecd04818b07c06d01f830f1a P b7094e9660dc47ceb5e9f70e089ef166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5438c5a3070f40d99bb7955dc412fbcc in term 0.
I20260812 11:10:51.364187 17568 leader_election.cc:302] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5438c5a3070f40d99bb7955dc412fbcc, b7094e9660dc47ceb5e9f70e089ef166; no voters:
I20260812 11:10:51.364737 17791 raft_consensus.cc:2781] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:10:51.364969 17791 raft_consensus.cc:482] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:10:51.365147 17791 raft_consensus.cc:3037] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:10:51.365621 17481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a99dd1ffecd04818b07c06d01f830f1a" candidate_uuid: "5438c5a3070f40d99bb7955dc412fbcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3884e2014802405c8f38db8f6dcabc25" is_pre_election: true
I20260812 11:10:51.366154 17481 raft_consensus.cc:2454] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5438c5a3070f40d99bb7955dc412fbcc in term 0.
I20260812 11:10:51.367983 17791 raft_consensus.cc:504] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } }
I20260812 11:10:51.368631 17791 leader_election.cc:288] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc [CANDIDATE]: Term 1 election: Requested vote from peers b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789), 3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959)
I20260812 11:10:51.369223 17270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a99dd1ffecd04818b07c06d01f830f1a" candidate_uuid: "5438c5a3070f40d99bb7955dc412fbcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7094e9660dc47ceb5e9f70e089ef166"
I20260812 11:10:51.369319 17481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a99dd1ffecd04818b07c06d01f830f1a" candidate_uuid: "5438c5a3070f40d99bb7955dc412fbcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3884e2014802405c8f38db8f6dcabc25"
I20260812 11:10:51.369537 17270 raft_consensus.cc:3037] T a99dd1ffecd04818b07c06d01f830f1a P b7094e9660dc47ceb5e9f70e089ef166 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:10:51.369630 17481 raft_consensus.cc:3037] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:10:51.372051 17270 raft_consensus.cc:2454] T a99dd1ffecd04818b07c06d01f830f1a P b7094e9660dc47ceb5e9f70e089ef166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5438c5a3070f40d99bb7955dc412fbcc in term 1.
I20260812 11:10:51.372138 17481 raft_consensus.cc:2454] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5438c5a3070f40d99bb7955dc412fbcc in term 1.
I20260812 11:10:51.372550 17568 leader_election.cc:302] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5438c5a3070f40d99bb7955dc412fbcc, b7094e9660dc47ceb5e9f70e089ef166; no voters:
I20260812 11:10:51.372961 17791 raft_consensus.cc:2781] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:10:51.373754 17791 raft_consensus.cc:686] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc [term 1 LEADER]: Becoming Leader. State: Replica: 5438c5a3070f40d99bb7955dc412fbcc, State: Running, Role: LEADER
I20260812 11:10:51.374282 17791 consensus_queue.cc:237] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } }
I20260812 11:10:51.379709 17053 catalog_manager.cc:5495] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131). New cstate: current_term: 1 leader_uuid: "5438c5a3070f40d99bb7955dc412fbcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } health_report { overall_health: HEALTHY } } }
W20260812 11:10:51.394199 17542 tablet.cc:2369] T 6dda8ac5b4254f1296f26326220247e2 P 3884e2014802405c8f38db8f6dcabc25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:10:51.414078 17798 raft_consensus.cc:482] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:10:51.414394 17798 raft_consensus.cc:504] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } }
I20260812 11:10:51.416268 17798 leader_election.cc:288] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427), 3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959)
I20260812 11:10:51.426118 17672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dda8ac5b4254f1296f26326220247e2" candidate_uuid: "b7094e9660dc47ceb5e9f70e089ef166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5438c5a3070f40d99bb7955dc412fbcc" is_pre_election: true
I20260812 11:10:51.426570 17672 raft_consensus.cc:2454] T 6dda8ac5b4254f1296f26326220247e2 P 5438c5a3070f40d99bb7955dc412fbcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7094e9660dc47ceb5e9f70e089ef166 in term 0.
I20260812 11:10:51.426708 17481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dda8ac5b4254f1296f26326220247e2" candidate_uuid: "b7094e9660dc47ceb5e9f70e089ef166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3884e2014802405c8f38db8f6dcabc25" is_pre_election: true
I20260812 11:10:51.427011 17481 raft_consensus.cc:2454] T 6dda8ac5b4254f1296f26326220247e2 P 3884e2014802405c8f38db8f6dcabc25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7094e9660dc47ceb5e9f70e089ef166 in term 0.
I20260812 11:10:51.427150 17170 leader_election.cc:302] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5438c5a3070f40d99bb7955dc412fbcc, b7094e9660dc47ceb5e9f70e089ef166; no voters:
I20260812 11:10:51.427772 17798 raft_consensus.cc:2781] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:10:51.427989 17798 raft_consensus.cc:482] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:10:51.428122 17798 raft_consensus.cc:3037] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:10:51.430773 17798 raft_consensus.cc:504] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } }
I20260812 11:10:51.431474 17798 leader_election.cc:288] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 [CANDIDATE]: Term 1 election: Requested vote from peers 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427), 3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959)
I20260812 11:10:51.431926 17672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dda8ac5b4254f1296f26326220247e2" candidate_uuid: "b7094e9660dc47ceb5e9f70e089ef166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5438c5a3070f40d99bb7955dc412fbcc"
I20260812 11:10:51.431991 17481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dda8ac5b4254f1296f26326220247e2" candidate_uuid: "b7094e9660dc47ceb5e9f70e089ef166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3884e2014802405c8f38db8f6dcabc25"
I20260812 11:10:51.432149 17672 raft_consensus.cc:3037] T 6dda8ac5b4254f1296f26326220247e2 P 5438c5a3070f40d99bb7955dc412fbcc [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:10:51.432312 17481 raft_consensus.cc:3037] T 6dda8ac5b4254f1296f26326220247e2 P 3884e2014802405c8f38db8f6dcabc25 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:10:51.434288 17672 raft_consensus.cc:2454] T 6dda8ac5b4254f1296f26326220247e2 P 5438c5a3070f40d99bb7955dc412fbcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7094e9660dc47ceb5e9f70e089ef166 in term 1.
I20260812 11:10:51.434665 17481 raft_consensus.cc:2454] T 6dda8ac5b4254f1296f26326220247e2 P 3884e2014802405c8f38db8f6dcabc25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7094e9660dc47ceb5e9f70e089ef166 in term 1.
I20260812 11:10:51.434777 17170 leader_election.cc:302] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5438c5a3070f40d99bb7955dc412fbcc, b7094e9660dc47ceb5e9f70e089ef166; no voters:
I20260812 11:10:51.435145 17798 raft_consensus.cc:2781] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:10:51.435887 17798 raft_consensus.cc:686] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 [term 1 LEADER]: Becoming Leader. State: Replica: b7094e9660dc47ceb5e9f70e089ef166, State: Running, Role: LEADER
I20260812 11:10:51.436455 17798 consensus_queue.cc:237] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } }
I20260812 11:10:51.442276 17053 catalog_manager.cc:5495] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129). New cstate: current_term: 1 leader_uuid: "b7094e9660dc47ceb5e9f70e089ef166" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } health_report { overall_health: HEALTHY } } }
I20260812 11:10:51.817863 17791 consensus_queue.cc:1035] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:10:51.826062 17791 consensus_queue.cc:1035] T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:10:51.890128 17798 consensus_queue.cc:1035] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:10:51.897516 17798 consensus_queue.cc:1035] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:10:51.928282 17787 consensus_queue.cc:1035] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:10:51.935930 17787 consensus_queue.cc:1035] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:10:53.492815 17344 maintenance_manager.cc:419] P b7094e9660dc47ceb5e9f70e089ef166: Scheduling FlushMRSOp(6dda8ac5b4254f1296f26326220247e2): perf score=1.000000
W20260812 11:10:53.708449 17742 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.16.151.129:42789, user_credentials={real_user=slave}} blocked reactor thread for 90973.6us
I20260812 11:10:53.783370 17738 maintenance_manager.cc:419] P 5438c5a3070f40d99bb7955dc412fbcc: Scheduling FlushMRSOp(6dda8ac5b4254f1296f26326220247e2): perf score=1.000000
I20260812 11:10:53.900576 17542 maintenance_manager.cc:419] P 3884e2014802405c8f38db8f6dcabc25: Scheduling FlushMRSOp(6dda8ac5b4254f1296f26326220247e2): perf score=1.000000
W20260812 11:10:54.336732 17742 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.16.151.130:37959, user_credentials={real_user=slave}} blocked reactor thread for 50056.3us
W20260812 11:10:54.391834 17572 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-2/data/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-2/data/data (496562176 bytes available vs 518351011 bytes reserved) (error 28)
W20260812 11:10:54.499599 17572 tablet_replica_mm_ops.cc:214] T 6dda8ac5b4254f1296f26326220247e2 P 5438c5a3070f40d99bb7955dc412fbcc: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: tablet 6dda8ac5b4254f1296f26326220247e2: unable to open output file for column float_val FLOAT NULLABLE: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 6dda8ac5b4254f1296f26326220247e2's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
F20260812 11:10:54.499928 17572 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: tablet 6dda8ac5b4254f1296f26326220247e2: unable to open output file for column float_val FLOAT NULLABLE: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 6dda8ac5b4254f1296f26326220247e2's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
*** Check failure stack trace: ***
@ 0x7fee6dba79ad google::LogMessage::Fail() at ??:0
@ 0x7fee6dbaa88a google::LogMessage::SendToLog() at ??:0
@ 0x7fee6dba74de google::LogMessage::Flush() at ??:0
@ 0x7fee6dba8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fee7cf7ce7d kudu::tablet::FlushMRSOp::Perform() at ??:0
@ 0x7fee70695090 kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7fee70697e53 kudu::MaintenanceManager::RunSchedulerThread()::$_3::operator()() at ??:0
@ 0x7fee70697ce9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fee707f3345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fee707f691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fee707f6799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fee707d1eaf kudu::Thread::SuperviseThread() at ??:0
W20260812 11:10:54.714036 17911 log.cc:924] Time spent T a99dd1ffecd04818b07c06d01f830f1a P 5438c5a3070f40d99bb7955dc412fbcc: Append to log took a long time: real 0.072s user 0.004s sys 0.000s
@ 0x7fee73aab6db start_thread at ??:0
@ 0x7fee6ca4161f clone at ??:0
W20260812 11:10:55.169175 17744 meta_cache.cc:302] tablet a99dd1ffecd04818b07c06d01f830f1a: replica 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427) has failed: Network error: recv got EOF from 127.16.151.131:38427 (error 108)
W20260812 11:10:55.179113 17215 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35748: Illegal state: replica b7094e9660dc47ceb5e9f70e089ef166 is not leader of this config: current role FOLLOWER
W20260812 11:10:55.186474 17371 connection.cc:535] client connection to 127.16.151.131:38427 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 11:10:55.186827 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 11:10:55.195124 17170 connection.cc:535] client connection to 127.16.151.131:38427 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 11:10:55.195387 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 11:10:55.224871 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 11:10:55.226311 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 11:10:55.445235 17956 log.cc:924] Time spent T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
W20260812 11:10:55.563692 17742 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.16.151.130:37959, user_credentials={real_user=slave}} blocked reactor thread for 63359.1us
I20260812 11:10:56.124565 17957 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:43750 (ReqId={client: 3d7a6e2a9c154d0a9045c3b86d180a94, seq_no=4, attempt_no=0}) took 1200 ms. Trace:
I20260812 11:10:56.124910 17957 rpcz_store.cc:276] 0812 11:10:54.923984 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 11:10:54.924079 (+ 95us) service_pool.cc:227] Handling call
0812 11:10:56.124472 (+1200393us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0812 11:10:54.934546 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88f0bbdf7cac496198e6b83299d50850
0812 11:10:54.934847 (+ 301us) write_op.cc:432] Acquiring schema lock in shared mode
0812 11:10:54.934855 (+ 8us) write_op.cc:435] Acquired schema lock
0812 11:10:54.934858 (+ 3us) tablet.cc:660] Decoding operations
0812 11:10:55.268015 (+333157us) write_op.cc:620] Acquiring the partition lock for write op
0812 11:10:55.268038 (+ 23us) write_op.cc:641] Partition lock acquired for write op
0812 11:10:55.268046 (+ 8us) tablet.cc:683] Acquiring locks for 8062 operations
0812 11:10:55.351156 (+ 83110us) tablet.cc:699] Row locks acquired
0812 11:10:55.351161 (+ 5us) write_op.cc:260] PREPARE: finished
0812 11:10:55.351257 (+ 96us) write_op.cc:270] Start()
0812 11:10:55.351312 (+ 55us) write_op.cc:276] Timestamp: P: 1786533055351178 usec, L: 0
0812 11:10:55.351316 (+ 4us) op_driver.cc:331] REPLICATION: starting
0812 11:10:55.353272 (+ 1956us) log.cc:842] Serialized 1258310 byte log entry
0812 11:10:55.453487 (+100215us) op_driver.cc:444] REPLICATION: finished
0812 11:10:55.461749 (+ 8262us) write_op.cc:301] APPLY: starting
0812 11:10:55.461774 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0812 11:10:55.501667 (+ 39893us) tablet.cc:1368] finished BulkCheckPresence
0812 11:10:55.501740 (+ 73us) tablet.cc:1370] starting ApplyRowOperation cycle
0812 11:10:56.099120 (+597380us) tablet.cc:1381] finished ApplyRowOperation cycle
0812 11:10:56.104558 (+ 5438us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 11:10:56.104572 (+ 14us) write_op.cc:312] APPLY: finished
0812 11:10:56.117991 (+ 13419us) log.cc:842] Serialized 48394 byte log entry
0812 11:10:56.118172 (+ 181us) write_op.cc:489] Releasing partition, row and schema locks
0812 11:10:56.123694 (+ 5522us) write_op.cc:454] Released schema lock
0812 11:10:56.124224 (+ 530us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":8169,"mutex_wait_us":768,"num_ops":8062,"prepare.queue_time_us":7496,"prepare.run_cpu_time_us":253804,"prepare.run_wall_time_us":490912,"raft.queue_time_us":38119,"raft.run_cpu_time_us":26708,"raft.run_wall_time_us":41034,"replication_time_us":102133,"thread_start_us":84222,"threads_started":4,"wal-append.queue_time_us":39456}]]}
I20260812 11:10:56.208519 17960 raft_consensus.cc:482] T a99dd1ffecd04818b07c06d01f830f1a P b7094e9660dc47ceb5e9f70e089ef166 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5438c5a3070f40d99bb7955dc412fbcc)
I20260812 11:10:56.208817 17960 raft_consensus.cc:504] T a99dd1ffecd04818b07c06d01f830f1a P b7094e9660dc47ceb5e9f70e089ef166 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } }
I20260812 11:10:56.209889 17960 leader_election.cc:288] T a99dd1ffecd04818b07c06d01f830f1a P b7094e9660dc47ceb5e9f70e089ef166 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959), 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427)
I20260812 11:10:56.211112 17481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a99dd1ffecd04818b07c06d01f830f1a" candidate_uuid: "b7094e9660dc47ceb5e9f70e089ef166" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3884e2014802405c8f38db8f6dcabc25" is_pre_election: true
W20260812 11:10:56.212546 17742 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.16.151.130:37959, user_credentials={real_user=slave}} blocked reactor thread for 85096.3us
W20260812 11:10:56.214169 17170 leader_election.cc:334] T a99dd1ffecd04818b07c06d01f830f1a P b7094e9660dc47ceb5e9f70e089ef166 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111)
I20260812 11:10:56.214388 17170 leader_election.cc:302] T a99dd1ffecd04818b07c06d01f830f1a P b7094e9660dc47ceb5e9f70e089ef166 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b7094e9660dc47ceb5e9f70e089ef166; no voters: 3884e2014802405c8f38db8f6dcabc25, 5438c5a3070f40d99bb7955dc412fbcc
I20260812 11:10:56.215212 17960 raft_consensus.cc:2726] T a99dd1ffecd04818b07c06d01f830f1a P b7094e9660dc47ceb5e9f70e089ef166 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:10:56.228969 17959 raft_consensus.cc:482] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5438c5a3070f40d99bb7955dc412fbcc)
I20260812 11:10:56.229252 17959 raft_consensus.cc:504] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } }
I20260812 11:10:56.230209 17959 leader_election.cc:288] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789), 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427)
I20260812 11:10:56.232182 17270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a99dd1ffecd04818b07c06d01f830f1a" candidate_uuid: "3884e2014802405c8f38db8f6dcabc25" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b7094e9660dc47ceb5e9f70e089ef166" is_pre_election: true
I20260812 11:10:56.232549 17270 raft_consensus.cc:2454] T a99dd1ffecd04818b07c06d01f830f1a P b7094e9660dc47ceb5e9f70e089ef166 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3884e2014802405c8f38db8f6dcabc25 in term 1.
I20260812 11:10:56.234258 17369 leader_election.cc:302] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3884e2014802405c8f38db8f6dcabc25, b7094e9660dc47ceb5e9f70e089ef166; no voters:
I20260812 11:10:56.235066 17959 raft_consensus.cc:2781] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 11:10:56.235219 17959 raft_consensus.cc:482] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5438c5a3070f40d99bb7955dc412fbcc)
I20260812 11:10:56.235344 17959 raft_consensus.cc:3037] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 [term 1 FOLLOWER]: Advancing to term 2
W20260812 11:10:56.237251 17371 leader_election.cc:334] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111)
I20260812 11:10:56.240411 17959 raft_consensus.cc:504] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } }
I20260812 11:10:56.242228 17270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a99dd1ffecd04818b07c06d01f830f1a" candidate_uuid: "3884e2014802405c8f38db8f6dcabc25" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b7094e9660dc47ceb5e9f70e089ef166"
I20260812 11:10:56.242488 17270 raft_consensus.cc:3037] T a99dd1ffecd04818b07c06d01f830f1a P b7094e9660dc47ceb5e9f70e089ef166 [term 1 FOLLOWER]: Advancing to term 2
I20260812 11:10:56.246451 17270 raft_consensus.cc:2454] T a99dd1ffecd04818b07c06d01f830f1a P b7094e9660dc47ceb5e9f70e089ef166 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3884e2014802405c8f38db8f6dcabc25 in term 2.
I20260812 11:10:56.247102 17369 leader_election.cc:302] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3884e2014802405c8f38db8f6dcabc25, b7094e9660dc47ceb5e9f70e089ef166; no voters:
I20260812 11:10:56.249400 17959 leader_election.cc:288] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 [CANDIDATE]: Term 2 election: Requested vote from peers b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789), 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427)
I20260812 11:10:56.250010 17959 raft_consensus.cc:2781] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 11:10:56.250483 17959 raft_consensus.cc:686] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 [term 2 LEADER]: Becoming Leader. State: Replica: 3884e2014802405c8f38db8f6dcabc25, State: Running, Role: LEADER
I20260812 11:10:56.251111 17959 consensus_queue.cc:237] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } }
W20260812 11:10:56.251570 17371 leader_election.cc:334] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111)
I20260812 11:10:56.257731 17051 catalog_manager.cc:5495] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 reported cstate change: term changed from 1 to 2, leader changed from 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131) to 3884e2014802405c8f38db8f6dcabc25 (127.16.151.130). New cstate: current_term: 2 leader_uuid: "3884e2014802405c8f38db8f6dcabc25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3884e2014802405c8f38db8f6dcabc25" member_type: VOTER last_known_addr { host: "127.16.151.130" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5438c5a3070f40d99bb7955dc412fbcc" member_type: VOTER last_known_addr { host: "127.16.151.131" port: 38427 } health_report { overall_health: UNKNOWN } } }
I20260812 11:10:56.514899 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:10:56.541157 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:10:56.591806 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 11:10:56.699429 17270 raft_consensus.cc:1261] T a99dd1ffecd04818b07c06d01f830f1a P b7094e9660dc47ceb5e9f70e089ef166 [term 2 FOLLOWER]: Refusing update from remote peer 3884e2014802405c8f38db8f6dcabc25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260812 11:10:56.700309 17977 consensus_queue.cc:1035] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7094e9660dc47ceb5e9f70e089ef166" member_type: VOTER last_known_addr { host: "127.16.151.129" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260812 11:10:56.717760 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 11:10:56.868500 17375 maintenance_manager.cc:643] P 3884e2014802405c8f38db8f6dcabc25: FlushMRSOp(6dda8ac5b4254f1296f26326220247e2) complete. Timing: real 2.967s user 1.294s sys 0.233s Metrics: {"bytes_written":436310,"cfile_init":1,"compiler_manager_pool.queue_time_us":19753,"compiler_manager_pool.run_cpu_time_us":1028617,"compiler_manager_pool.run_wall_time_us":1969540,"dirs.queue_time_us":499,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":1067,"drs_written":1,"lbm_read_time_us":140,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5163,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"spinlock_wait_cycles":247680,"thread_start_us":26420,"threads_started":3,"wal-append.queue_time_us":16766}
I20260812 11:10:56.870615 17542 maintenance_manager.cc:419] P 3884e2014802405c8f38db8f6dcabc25: Scheduling FlushMRSOp(88f0bbdf7cac496198e6b83299d50850): perf score=1.937278
I20260812 11:10:56.871181 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:10:56.887749 17173 maintenance_manager.cc:643] P b7094e9660dc47ceb5e9f70e089ef166: FlushMRSOp(6dda8ac5b4254f1296f26326220247e2) complete. Timing: real 3.394s user 1.314s sys 0.223s Metrics: {"bytes_written":436310,"cfile_init":1,"compiler_manager_pool.queue_time_us":7882,"compiler_manager_pool.run_cpu_time_us":1080094,"compiler_manager_pool.run_wall_time_us":2777095,"dirs.queue_time_us":936,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":983,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6184,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":23346,"threads_started":3,"wal-append.queue_time_us":10319}
I20260812 11:10:56.890241 17344 maintenance_manager.cc:419] P b7094e9660dc47ceb5e9f70e089ef166: Scheduling UndoDeltaBlockGCOp(6dda8ac5b4254f1296f26326220247e2): 62918 bytes on disk
I20260812 11:10:56.891389 17173 maintenance_manager.cc:643] P b7094e9660dc47ceb5e9f70e089ef166: UndoDeltaBlockGCOp(6dda8ac5b4254f1296f26326220247e2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20260812 11:10:56.892360 17344 maintenance_manager.cc:419] P b7094e9660dc47ceb5e9f70e089ef166: Scheduling FlushMRSOp(88f0bbdf7cac496198e6b83299d50850): perf score=1.937278
I20260812 11:10:56.894112 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:10:56.965188 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 11:10:57.317175 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:10:57.341029 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:10:57.398645 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:10:57.417271 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260812 11:10:57.672964 17173 maintenance_manager.cc:643] P b7094e9660dc47ceb5e9f70e089ef166: FlushMRSOp(88f0bbdf7cac496198e6b83299d50850) complete. Timing: real 0.780s user 0.684s sys 0.079s Metrics: {"bytes_written":472920,"cfile_init":1,"dirs.queue_time_us":423,"dirs.run_cpu_time_us":288,"dirs.run_wall_time_us":1607,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3596,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":506,"threads_started":2,"wal-append.queue_time_us":395}
I20260812 11:10:57.673789 17344 maintenance_manager.cc:419] P b7094e9660dc47ceb5e9f70e089ef166: Scheduling FlushMRSOp(a99dd1ffecd04818b07c06d01f830f1a): perf score=1.937278
I20260812 11:10:57.703025 17375 maintenance_manager.cc:643] P 3884e2014802405c8f38db8f6dcabc25: FlushMRSOp(88f0bbdf7cac496198e6b83299d50850) complete. Timing: real 0.832s user 0.696s sys 0.127s Metrics: {"bytes_written":472920,"cfile_init":1,"dirs.queue_time_us":432,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":1385,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3624,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"spinlock_wait_cycles":4480,"thread_start_us":575,"threads_started":2,"wal-append.queue_time_us":393}
I20260812 11:10:57.704027 17542 maintenance_manager.cc:419] P 3884e2014802405c8f38db8f6dcabc25: Scheduling FlushMRSOp(a99dd1ffecd04818b07c06d01f830f1a): perf score=1.937278
I20260812 11:10:57.902248 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:10:57.915441 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:10:57.962127 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:10:57.988969 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260812 11:10:58.638715 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:10:58.648356 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260812 11:10:58.653956 17173 maintenance_manager.cc:643] P b7094e9660dc47ceb5e9f70e089ef166: FlushMRSOp(a99dd1ffecd04818b07c06d01f830f1a) complete. Timing: real 0.980s user 0.881s sys 0.077s Metrics: {"bytes_written":471972,"cfile_init":1,"dirs.queue_time_us":472,"dirs.run_cpu_time_us":238,"dirs.run_wall_time_us":1453,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4305,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":632,"threads_started":2,"wal-append.queue_time_us":480}
I20260812 11:10:58.654835 17344 maintenance_manager.cc:419] P b7094e9660dc47ceb5e9f70e089ef166: Scheduling UndoDeltaBlockGCOp(a99dd1ffecd04818b07c06d01f830f1a): 63433 bytes on disk
I20260812 11:10:58.655490 17173 maintenance_manager.cc:643] P b7094e9660dc47ceb5e9f70e089ef166: UndoDeltaBlockGCOp(a99dd1ffecd04818b07c06d01f830f1a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20260812 11:10:58.656275 17344 maintenance_manager.cc:419] P b7094e9660dc47ceb5e9f70e089ef166: Scheduling UndoDeltaBlockGCOp(88f0bbdf7cac496198e6b83299d50850): 63419 bytes on disk
I20260812 11:10:58.656958 17173 maintenance_manager.cc:643] P b7094e9660dc47ceb5e9f70e089ef166: UndoDeltaBlockGCOp(88f0bbdf7cac496198e6b83299d50850) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20260812 11:10:58.661907 17375 maintenance_manager.cc:643] P 3884e2014802405c8f38db8f6dcabc25: FlushMRSOp(a99dd1ffecd04818b07c06d01f830f1a) complete. Timing: real 0.958s user 0.872s sys 0.076s Metrics: {"bytes_written":471972,"cfile_init":1,"dirs.queue_time_us":517,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":1260,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3347,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":825,"threads_started":2,"wal-append.queue_time_us":574}
I20260812 11:10:58.662714 17542 maintenance_manager.cc:419] P 3884e2014802405c8f38db8f6dcabc25: Scheduling UndoDeltaBlockGCOp(a99dd1ffecd04818b07c06d01f830f1a): 63433 bytes on disk
I20260812 11:10:58.663645 17375 maintenance_manager.cc:643] P 3884e2014802405c8f38db8f6dcabc25: UndoDeltaBlockGCOp(a99dd1ffecd04818b07c06d01f830f1a) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20260812 11:10:58.664258 17542 maintenance_manager.cc:419] P 3884e2014802405c8f38db8f6dcabc25: Scheduling UndoDeltaBlockGCOp(88f0bbdf7cac496198e6b83299d50850): 63419 bytes on disk
I20260812 11:10:58.664740 17375 maintenance_manager.cc:643] P 3884e2014802405c8f38db8f6dcabc25: UndoDeltaBlockGCOp(88f0bbdf7cac496198e6b83299d50850) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20260812 11:10:58.665326 17542 maintenance_manager.cc:419] P 3884e2014802405c8f38db8f6dcabc25: Scheduling UndoDeltaBlockGCOp(6dda8ac5b4254f1296f26326220247e2): 62918 bytes on disk
I20260812 11:10:58.665879 17375 maintenance_manager.cc:643] P 3884e2014802405c8f38db8f6dcabc25: UndoDeltaBlockGCOp(6dda8ac5b4254f1296f26326220247e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:10:58.684921 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:10:59.164081 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260812 11:10:59.530529 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:10:59.532948 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:10:59.576908 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:00.016193 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260812 11:11:00.619016 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:11:00.620059 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:11:00.620704 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260812 11:11:00.667755 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:00.667886 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:00.704913 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:01.742007 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260812 11:11:01.797551 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:01.799687 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:01.843278 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:02.747308 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260812 11:11:02.933300 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:02.935472 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:02.981801 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:03.006927 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260812 11:11:04.073333 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260812 11:11:04.076231 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.129:42789, tserver@127.16.151.130:37959
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:04.110298 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:04.379185 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260812 11:11:04.981528 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260812 11:11:05.200263 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:05.201530 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:05.233662 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:05.697536 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:11:05.699061 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260812 11:11:05.949548 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20260812 11:11:06.321689 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:06.322466 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:06.355669 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:07.034538 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260812 11:11:07.438982 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260812 11:11:07.446744 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:07.449381 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:07.481571 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:08.390985 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260812 11:11:08.566596 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:08.567713 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:08.604396 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:09.480664 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260812 11:11:09.690493 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:09.693006 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:09.724813 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:09.814816 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260812 11:11:10.812107 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:10.813532 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:10.849501 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:11.058343 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:11:11.059499 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260812 11:11:11.201150 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20260812 11:11:11.944314 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:11.946499 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:11.979418 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:12.118387 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260812 11:11:12.139045 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260812 11:11:13.074273 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260812 11:11:13.074772 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.129:42789, tserver@127.16.151.130:37959
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:13.111915 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:13.895848 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260812 11:11:14.198179 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:14.199687 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:14.232366 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:14.484445 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260812 11:11:14.724181 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260812 11:11:15.324152 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:15.325291 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:15.369946 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:16.212742 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:11:16.213871 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260812 11:11:16.458676 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20260812 11:11:16.470268 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260812 11:11:16.470407 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:16.508150 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:17.077577 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260812 11:11:17.124770 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260812 11:11:17.598389 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:17.601778 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:17.632086 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 11:11:18.733819 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:18.733865 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.129:42789, tserver@127.16.151.130:37959
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:18.771687 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:18.857133 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260812 11:11:19.548835 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260812 11:11:19.616662 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260812 11:11:19.870100 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:19.871634 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:19.910929 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 11:11:21.001613 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:21.003857 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:21.039970 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:21.325466 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:11:21.326447 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260812 11:11:21.464113 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:11:21.919020 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260812 11:11:22.132128 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:22.136781 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:22.180120 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:22.275281 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260812 11:11:23.268714 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:23.271826 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:23.303226 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:23.890482 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260812 11:11:24.389099 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:24.396798 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:24.434254 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:24.609192 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260812 11:11:24.908568 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260812 11:11:25.522802 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:25.523228 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:25.563024 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:26.198323 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260812 11:11:26.511842 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20260812 11:11:26.651345 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:26.652938 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:26.686371 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:26.741883 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:11:26.949174 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260812 11:11:27.539690 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260812 11:11:27.771397 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:27.777030 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:27.820258 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:28.698441 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260812 11:11:28.917613 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260812 11:11:28.917613 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.129:42789, tserver@127.16.151.130:37959
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:28.953717 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:29.406056 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260812 11:11:30.043296 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:30.045380 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20260812 11:11:30.053412 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:30.077982 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 11:11:31.176779 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:31.177218 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:31.218396 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:31.275876 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260812 11:11:31.642302 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:11:32.084723 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260812 11:11:32.256130 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260812 11:11:32.257637 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20260812 11:11:32.307876 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260812 11:11:32.307958 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.129:42789, tserver@127.16.151.130:37959
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:32.341248 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 11:11:33.515460 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:33.517273 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:33.595201 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:33.844513 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260812 11:11:34.608021 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20260812 11:11:34.691282 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:34.695503 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
W20260812 11:11:34.735445 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:34.739691 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 11:11:35.830336 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:35.833040 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:35.867074 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:36.360642 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20260812 11:11:36.955794 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260812 11:11:36.956211 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.129:42789, tserver@127.16.151.130:37959
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:36.995869 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:37.054549 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260812 11:11:37.055639 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20260812 11:11:37.173434 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20260812 11:11:37.446946 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260812 11:11:38.088093 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260812 11:11:38.088127 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:38.128331 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:38.943593 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20260812 11:11:39.226199 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:39.226998 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:39.277726 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:39.626276 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20260812 11:11:39.954137 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20260812 11:11:40.372231 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:40.374627 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:40.407624 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:41.460664 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20260812 11:11:41.507952 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:41.507972 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.129:42789, tserver@127.16.151.130:37959
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:41.555397 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:41.960280 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20260812 11:11:42.444135 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:11:42.537384 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20260812 11:11:42.641656 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:42.645380 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:42.678752 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:42.775610 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:11:43.725281 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20260812 11:11:43.772853 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:43.774521 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:43.813365 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:44.671839 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20260812 11:11:44.909257 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:44.909413 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:44.955863 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:45.140189 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20260812 11:11:45.882210 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20260812 11:11:46.055588 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260812 11:11:46.055668 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.129:42789, tserver@127.16.151.130:37959
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:46.103008 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 11:11:47.194267 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:47.195068 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:47.228452 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:47.251097 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20260812 11:11:47.367069 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20260812 11:11:47.777127 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:11:48.280702 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20260812 11:11:48.282227 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20260812 11:11:48.317839 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:48.319049 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:48.354260 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 11:11:49.463505 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:49.464704 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20260812 11:11:49.472047 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:49.498905 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:50.071216 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20260812 11:11:50.591797 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:50.593065 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:50.629096 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:50.836910 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20260812 11:11:51.725759 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:51.732072 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:51.771332 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:52.020368 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20260812 11:11:52.754184 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20260812 11:11:52.862336 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260812 11:11:52.862396 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20260812 11:11:52.870851 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:52.906250 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:53.283003 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:11:53.284472 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20260812 11:11:53.997128 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:54.000332 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:54.047093 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:54.415272 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20260812 11:11:55.150478 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:55.157374 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20260812 11:11:55.172978 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:55.195163 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:55.781194 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20260812 11:11:56.281698 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:56.282737 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:56.317759 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:56.631774 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20260812 11:11:57.407660 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:57.408540 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:57.454777 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:57.687913 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20260812 11:11:57.967218 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20260812 11:11:58.205631 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20260812 11:11:58.545049 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:58.545581 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20260812 11:11:58.578861 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:58.589720 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:11:59.291594 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20260812 11:11:59.683365 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:11:59.697902 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:11:59.736093 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:00.045507 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20260812 11:12:00.754096 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20260812 11:12:00.832823 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:00.832916 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:00.869854 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:01.799657 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20260812 11:12:01.963352 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:01.969862 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:02.003767 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:02.921083 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20260812 11:12:03.115552 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260812 11:12:03.119030 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.129:42789, tserver@127.16.151.130:37959
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:03.153268 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:03.243901 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20260812 11:12:03.352907 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:12:03.838397 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260812 11:12:04.246173 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260812 11:12:04.246214 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.129:42789, tserver@127.16.151.130:37959
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:04.279838 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:04.288491 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20260812 11:12:05.375667 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:05.377522 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20260812 11:12:05.433403 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:05.441084 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:05.906584 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20260812 11:12:06.530985 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260812 11:12:06.531060 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.129:42789, tserver@127.16.151.130:37959
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:06.566708 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:06.615670 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20260812 11:12:07.667223 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:07.667455 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.129:42789, tserver@127.16.151.130:37959
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:07.713873 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:07.997829 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20260812 11:12:08.246393 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20260812 11:12:08.398980 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20260812 11:12:08.813112 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:08.813210 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.129:42789, tserver@127.16.151.130:37959
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:08.848321 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:09.157410 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20260812 11:12:09.266168 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20260812 11:12:09.938318 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:09.940018 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:09.972298 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:10.214931 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20260812 11:12:10.984091 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20260812 11:12:11.065266 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260812 11:12:11.065670 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.129:42789, tserver@127.16.151.130:37959
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:11.111657 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:11.798856 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20260812 11:12:12.211239 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260812 11:12:12.211524 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.129:42789, tserver@127.16.151.130:37959
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:12.253394 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:12.541463 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20260812 11:12:13.344710 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:13.350886 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:13.384114 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:13.385963 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20260812 11:12:13.431242 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:12:14.452409 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20260812 11:12:14.485911 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:14.487944 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:14.531631 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:14.689072 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:12:15.120653 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20260812 11:12:15.643198 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260812 11:12:15.643273 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.129:42789, tserver@127.16.151.130:37959
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:15.690341 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:15.725651 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20260812 11:12:16.783443 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:16.784161 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:16.827628 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:16.881011 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20260812 11:12:17.787133 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20260812 11:12:17.925798 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:17.932711 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:17.976747 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:18.325181 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20260812 11:12:18.767033 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20260812 11:12:19.071121 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:19.071671 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:19.113750 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:19.327596 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20260812 11:12:19.803933 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260812 11:12:20.214090 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:20.214239 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.129:42789, tserver@127.16.151.130:37959
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:20.256789 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:20.352679 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20260812 11:12:20.925128 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20260812 11:12:21.342677 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:21.344987 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:21.378094 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:21.952745 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20260812 11:12:22.471451 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260812 11:12:22.471534 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.129:42789, tserver@127.16.151.130:37959
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:22.534296 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:23.107153 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20260812 11:12:23.564632 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20260812 11:12:23.634181 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:23.634640 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:23.680596 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:24.067896 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:12:24.651067 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20260812 11:12:24.779954 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:24.786361 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:24.829185 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:25.249045 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:12:25.861931 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20260812 11:12:25.926185 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:25.927790 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:25.966845 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:26.251827 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20260812 11:12:27.070485 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:27.070899 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:27.111711 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:27.464946 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20260812 11:12:28.128738 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20260812 11:12:28.215867 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:28.220666 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:28.254808 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:28.776787 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20260812 11:12:29.092031 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20260812 11:12:29.343649 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:29.346730 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:29.385555 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:29.957207 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20260812 11:12:30.300016 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260812 11:12:30.476258 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:30.477216 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:30.512545 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:30.597195 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20260812 11:12:31.254052 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20260812 11:12:31.615195 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:31.617501 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:31.652489 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:32.686826 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20260812 11:12:32.745136 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:32.745824 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:32.788738 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:33.187307 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20260812 11:12:33.829588 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20260812 11:12:33.878459 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:33.880355 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:33.914707 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:34.289397 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:12:34.794461 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20260812 11:12:35.003448 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:35.004016 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:35.037297 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:35.316231 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:12:35.563485 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20260812 11:12:36.133915 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:36.137766 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:36.181296 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:36.453189 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20260812 11:12:37.190954 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20260812 11:12:37.278785 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:37.285604 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:37.329191 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:38.096244 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20260812 11:12:38.422314 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:38.422963 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:38.472609 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:38.853498 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20260812 11:12:39.558725 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:39.563704 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:39.603312 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:39.630208 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:12:39.671794 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20260812 11:12:40.514885 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:12:40.632865 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20260812 11:12:40.708187 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:40.708848 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.129:42789, tserver@127.16.151.130:37959
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:40.744261 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:41.296267 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20260812 11:12:41.837098 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:41.841459 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:41.884256 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:42.206043 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20260812 11:12:42.830543 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20260812 11:12:42.979024 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:42.981724 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:43.028265 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:43.511142 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20260812 11:12:44.115590 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:44.116588 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:44.164566 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:44.631229 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:12:44.632447 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20260812 11:12:45.255626 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:45.255756 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:45.289031 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:45.391503 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20260812 11:12:45.605685 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:12:46.134775 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20260812 11:12:46.375033 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:46.377995 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:46.409040 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:47.173830 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20260812 11:12:47.499655 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:47.500110 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:47.533046 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:47.938913 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20260812 11:12:48.632468 17442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 11:12:48.633991 17241 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ad68f6c35c69479391bebe4dcd9e223c | 127.16.151.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.16.151.148:37557 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3884e2014802405c8f38db8f6dcabc25 | 127.16.151.130:37959 | HEALTHY | <none> | 2 | 0
b7094e9660dc47ceb5e9f70e089ef166 | 127.16.151.129:42789 | HEALTHY | <none> | 1 | 0
5438c5a3070f40d99bb7955dc412fbcc | 127.16.151.131:38427 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.16.151.131:38427: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.16.151.130:37959, 127.16.151.129:42789
ipki_server_key_size | 768 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.129 | experimental | 127.16.151.129:42789
local_ip_for_outbound_sockets | 127.16.151.130 | experimental | 127.16.151.130:37959
never_fsync | true | unsafe,advanced | 127.16.151.130:37959, 127.16.151.129:42789
openssl_security_level_override | 0 | unsafe,hidden | 127.16.151.130:37959, 127.16.151.129:42789
rpc_server_allow_ephemeral_ports | true | unsafe | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_format | pb | hidden | 127.16.151.130:37959, 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.151.129:42789
server_dump_info_path | /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.151.130:37959
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.16.151.130:37959, 127.16.151.129:42789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.16.151.148:37557 | 127.16.151.130:37959, 127.16.151.129:42789
time_source | builtin | 127.16.151.130:37959, 127.16.151.129:42789
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.16.151.190:43065, tserver@127.16.151.130:37959, tserver@127.16.151.129:42789
Tablet Summary
Tablet 6dda8ac5b4254f1296f26326220247e2 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING [LEADER]
All reported replicas are:
A = 5438c5a3070f40d99bb7955dc412fbcc
B = 3884e2014802405c8f38db8f6dcabc25
C = b7094e9660dc47ceb5e9f70e089ef166
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet a99dd1ffecd04818b07c06d01f830f1a of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 88f0bbdf7cac496198e6b83299d50850 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
b7094e9660dc47ceb5e9f70e089ef166 (127.16.151.129:42789): RUNNING
3884e2014802405c8f38db8f6dcabc25 (127.16.151.130:37959): RUNNING [LEADER]
5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): TS unavailable
All reported replicas are:
A = b7094e9660dc47ceb5e9f70e089ef166
B = 3884e2014802405c8f38db8f6dcabc25
C = 5438c5a3070f40d99bb7955dc412fbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 11:12:48.672644 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 11:12:48.710448 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20260812 11:12:49.027841 17036 maintenance_manager.cc:419] P ad68f6c35c69479391bebe4dcd9e223c: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033363
W20260812 11:12:49.031829 17034 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-tasknusDu0/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786533048094313-16990-0/minicluster-data/master-0/data/data (491425792 bytes available vs 518351011 bytes reserved) (error 28)
W20260812 11:12:49.032872 17034 tablet_replica_mm_ops.cc:214] T 00000000000000000000000000000000 P ad68f6c35c69479391bebe4dcd9e223c: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: tablet 00000000000000000000000000000000: unable to open output file for column entry_type INT8 NOT NULL: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 00000000000000000000000000000000's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
F20260812 11:12:49.033028 17034 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: tablet 00000000000000000000000000000000: unable to open output file for column entry_type INT8 NOT NULL: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 00000000000000000000000000000000's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
*** Check failure stack trace: ***
@ 0x7ff6891c39ad google::LogMessage::Fail() at ??:0
@ 0x7ff6891c688a google::LogMessage::SendToLog() at ??:0
@ 0x7ff6891c34de google::LogMessage::Flush() at ??:0
@ 0x7ff6891c4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff698598e7d kudu::tablet::FlushMRSOp::Perform() at ??:0
@ 0x7ff68bcb1090 kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7ff68bcb3e53 kudu::MaintenanceManager::RunSchedulerThread()::$_3::operator()() at ??:0
@ 0x7ff68bcb3ce9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff68be0f345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff68be1291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff68be12799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff68bdedeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff68f0c76db start_thread at ??:0
@ 0x7ff68805d61f clone at ??:0
W20260812 11:12:49.967502 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:12:49.968564 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20260812 11:12:50.014542 17343 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:12:50.014544 17541 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:12:50.639345 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:12:50.663908 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20260812 11:12:51.150421 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20260812 11:12:52.682855 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20260812 11:12:53.608477 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260812 11:12:53.771749 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260812 11:12:55.014827 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260812 11:12:55.015925 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260812 11:12:55.895977 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:12:55.896991 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260812 11:12:56.227445 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260812 11:12:57.427598 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260812 11:12:58.150465 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260812 11:12:58.750067 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260812 11:12:59.710402 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260812 11:13:00.173193 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:13:00.863368 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260812 11:13:01.378229 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:13:01.403096 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260812 11:13:01.974359 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260812 11:13:03.191900 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260812 11:13:03.824623 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260812 11:13:04.557125 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260812 11:13:05.452291 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:13:05.796763 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260812 11:13:06.643762 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260812 11:13:06.787487 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:13:07.113842 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260812 11:13:08.128787 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260812 11:13:09.408219 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260812 11:13:09.523547 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260812 11:13:10.567036 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260812 11:13:10.635038 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260812 11:13:11.913241 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260812 11:13:12.000813 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260812 11:13:12.320684 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:13:13.398777 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260812 11:13:14.329424 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260812 11:13:14.354050 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260812 11:13:15.598809 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260812 11:13:15.859259 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260812 11:13:16.487392 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260812 11:13:16.775497 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260812 11:13:17.408265 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:13:18.349086 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260812 11:13:19.004359 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260812 11:13:19.099058 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260812 11:13:20.705461 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260812 11:13:20.774871 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260812 11:13:21.325932 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260812 11:13:21.471755 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260812 11:13:22.672575 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:13:23.148181 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260812 11:13:23.607995 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260812 11:13:24.259418 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260812 11:13:25.812642 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260812 11:13:26.142369 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:13:26.143438 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260812 11:13:26.900671 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260812 11:13:28.202903 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:13:28.204276 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260812 11:13:28.751119 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260812 11:13:29.335908 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260812 11:13:30.519917 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260812 11:13:31.215580 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:13:31.216747 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260812 11:13:31.825785 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260812 11:13:33.096176 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260812 11:13:33.720351 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:13:33.841619 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260812 11:13:34.178185 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260812 11:13:35.474079 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260812 11:13:36.234643 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:13:36.436159 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260812 11:13:36.697613 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260812 11:13:38.152801 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260812 11:13:39.056109 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:13:39.180177 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260812 11:13:39.233469 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260812 11:13:40.526630 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260812 11:13:41.253795 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:13:41.603854 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260812 11:13:41.813735 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260812 11:13:43.289197 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260812 11:13:44.041244 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260812 11:13:44.173605 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260812 11:13:44.182090 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:13:45.640594 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260812 11:13:46.686479 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:13:46.687770 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260812 11:13:46.698118 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260812 11:13:48.225459 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260812 11:13:49.022260 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260812 11:13:49.105998 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260812 11:13:49.610993 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.16.151.190:43065) | 127.16.151.190:43065 | UNAVAILABLE
Error from 127.16.151.190:43065: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.16.151.190:43065, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
FAILED
I20260812 11:13:50.022418 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.16.151.190:43065, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:13:50.104298 17343 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (60 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:13:50.104560 17541 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (60 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:13:50.618388 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260812 11:13:51.372720 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260812 11:13:51.680302 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260812 11:13:51.906239 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260812 11:13:53.198369 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260812 11:13:54.016902 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260812 11:13:54.115976 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260812 11:13:54.645102 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:13:55.548964 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260812 11:13:56.484902 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260812 11:13:56.507591 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260812 11:13:57.010164 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:13:58.022094 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260812 11:13:58.788182 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260812 11:13:58.868938 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260812 11:13:59.985497 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:14:00.382944 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260812 11:14:01.263015 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260812 11:14:01.434408 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260812 11:14:02.140617 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260812 11:14:02.831576 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260812 11:14:03.724161 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260812 11:14:03.744526 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260812 11:14:05.275715 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:14:05.277100 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260812 11:14:06.291858 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260812 11:14:06.365893 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260812 11:14:07.370012 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:14:07.720603 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260812 11:14:08.764943 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260812 11:14:09.012403 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260812 11:14:10.294920 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:14:10.296424 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260812 11:14:11.412011 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260812 11:14:11.446410 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260812 11:14:12.418485 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:14:12.890026 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260812 11:14:13.797139 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260812 11:14:13.893576 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260812 11:14:15.641323 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:14:15.642954 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260812 11:14:16.479523 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260812 11:14:16.630826 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260812 11:14:17.611658 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:14:18.219965 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260812 11:14:19.153862 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260812 11:14:19.198995 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260812 11:14:20.839160 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:14:20.840469 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260812 11:14:21.677021 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260812 11:14:21.852694 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260812 11:14:22.623208 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:14:23.466843 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260812 11:14:24.018517 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260812 11:14:24.019313 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260812 11:14:26.083734 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:14:26.085278 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260812 11:14:26.458299 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260812 11:14:26.623677 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260812 11:14:27.686141 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:14:28.654987 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260812 11:14:28.730973 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260812 11:14:29.243677 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260812 11:14:30.941836 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260812 11:14:31.147090 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260812 11:14:31.373595 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:14:31.836822 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260812 11:14:32.763106 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:14:33.530503 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260812 11:14:33.540462 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260812 11:14:34.093226 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260812 11:14:36.020015 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260812 11:14:36.292850 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260812 11:14:36.702185 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260812 11:14:36.854862 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:14:38.043895 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:14:38.642621 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260812 11:14:38.921738 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260812 11:14:39.065024 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260812 11:14:41.367206 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260812 11:14:41.525620 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260812 11:14:41.604923 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260812 11:14:42.101218 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:14:43.317967 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:14:43.874176 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260812 11:14:43.930840 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260812 11:14:44.064574 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260812 11:14:46.354498 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260812 11:14:46.360773 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260812 11:14:46.370191 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260812 11:14:47.252054 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:14:48.380201 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:14:48.634758 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260812 11:14:48.773244 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260812 11:14:48.871599 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260812 11:14:50.194785 17343 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (120 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:14:50.194766 17541 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (120 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:14:51.195456 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260812 11:14:51.249401 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260812 11:14:51.320179 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.16.151.190:43065) | 127.16.151.190:43065 | UNAVAILABLE
Error from 127.16.151.190:43065: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.16.151.190:43065, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
FAILED
I20260812 11:14:51.322067 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.16.151.190:43065, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:14:52.821630 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:14:53.538437 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:14:53.727511 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260812 11:14:53.877434 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260812 11:14:54.078692 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260812 11:14:56.231736 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260812 11:14:56.435511 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260812 11:14:56.613114 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260812 11:14:57.915338 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:14:58.582785 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:14:58.961898 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260812 11:14:58.984043 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260812 11:14:59.027161 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260812 11:15:01.430397 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260812 11:15:01.439684 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260812 11:15:01.507507 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260812 11:15:03.228303 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:15:03.879251 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:15:03.880371 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260812 11:15:03.891327 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260812 11:15:04.284164 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260812 11:15:05.979091 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260812 11:15:06.473435 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260812 11:15:06.979475 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260812 11:15:08.523725 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:15:08.678649 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260812 11:15:08.907440 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:15:08.908545 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260812 11:15:09.648381 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260812 11:15:10.966637 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260812 11:15:11.313642 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260812 11:15:11.935436 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260812 11:15:13.597098 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260812 11:15:13.652477 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:15:14.038333 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:15:14.225324 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260812 11:15:14.230208 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260812 11:15:16.027011 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260812 11:15:16.656466 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260812 11:15:16.775969 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260812 11:15:18.787860 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260812 11:15:19.131469 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:15:19.132786 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260812 11:15:19.325721 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:15:19.327179 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260812 11:15:21.390875 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260812 11:15:21.540284 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260812 11:15:21.650542 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260812 11:15:23.911136 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260812 11:15:23.914222 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260812 11:15:24.078128 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260812 11:15:24.458706 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:15:24.500730 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:15:26.317507 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260812 11:15:26.353449 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260812 11:15:26.782827 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260812 11:15:28.901768 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260812 11:15:28.975713 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260812 11:15:29.518849 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:15:29.519786 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260812 11:15:29.841331 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:15:31.278496 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260812 11:15:31.573966 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260812 11:15:31.939038 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260812 11:15:33.760474 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260812 11:15:34.042196 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260812 11:15:34.357898 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260812 11:15:34.561991 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:15:35.032882 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:15:35.974725 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260812 11:15:36.615296 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260812 11:15:36.744417 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260812 11:15:38.456128 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260812 11:15:39.253842 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260812 11:15:39.473789 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260812 11:15:39.865829 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:15:40.250347 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:15:41.150156 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260812 11:15:41.882367 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260812 11:15:41.930581 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260812 11:15:43.914796 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260812 11:15:44.437958 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260812 11:15:44.462106 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260812 11:15:45.021070 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:15:45.511225 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:15:46.586091 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260812 11:15:47.044135 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260812 11:15:47.059263 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260812 11:15:48.967013 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260812 11:15:49.219129 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260812 11:15:49.595738 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260812 11:15:50.120747 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:15:50.284580 17541 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (180 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:15:50.284868 17343 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (180 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:15:50.923498 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:15:51.320117 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260812 11:15:51.468986 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260812 11:15:52.027760 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.16.151.190:43065) | 127.16.151.190:43065 | UNAVAILABLE
Error from 127.16.151.190:43065: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.16.151.190:43065, num_attempts: 343) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
FAILED
I20260812 11:15:52.346220 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.16.151.190:43065, num_attempts: 343) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:15:53.721465 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20260812 11:15:54.098762 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20260812 11:15:54.452160 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20260812 11:15:54.579537 23112 consensus_queue.cc:579] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 [LEADER]: Leader has been unable to successfully communicate with peer 5438c5a3070f40d99bb7955dc412fbcc for more than 300 seconds (300.011s)
I20260812 11:15:54.749416 23123 consensus_queue.cc:579] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 [LEADER]: Leader has been unable to successfully communicate with peer 5438c5a3070f40d99bb7955dc412fbcc for more than 300 seconds (300.094s)
W20260812 11:15:55.407454 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260812 11:15:56.268333 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:15:56.269680 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260812 11:15:56.511224 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20260812 11:15:56.593864 23112 consensus_queue.cc:579] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 [LEADER]: Leader has been unable to successfully communicate with peer 5438c5a3070f40d99bb7955dc412fbcc for more than 300 seconds (300.341s)
W20260812 11:15:57.143290 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260812 11:15:58.609187 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260812 11:15:59.316754 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260812 11:15:59.428998 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260812 11:16:00.536319 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:16:00.850333 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260812 11:16:01.710449 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20260812 11:16:01.926576 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260812 11:16:01.990538 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260812 11:16:03.172108 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260812 11:16:04.513598 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260812 11:16:04.596199 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260812 11:16:05.546027 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:16:05.657994 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260812 11:16:07.056577 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260812 11:16:07.201082 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:16:07.237708 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260812 11:16:08.356151 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260812 11:16:09.716357 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260812 11:16:09.934645 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260812 11:16:10.704655 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260812 11:16:10.742637 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260812 11:16:12.253212 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:16:12.369535 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260812 11:16:12.429255 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260812 11:16:13.043779 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260812 11:16:14.716086 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260812 11:16:14.821272 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260812 11:16:15.757611 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260812 11:16:15.891263 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:16:17.255002 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260812 11:16:17.412388 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260812 11:16:17.475901 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:16:18.479875 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260812 11:16:19.675125 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260812 11:16:19.795358 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260812 11:16:21.089550 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260812 11:16:21.132966 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:16:22.118218 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260812 11:16:22.373176 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260812 11:16:22.589511 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:16:23.479842 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260812 11:16:24.573377 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260812 11:16:25.045953 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260812 11:16:26.001618 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260812 11:16:26.285204 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:16:27.089510 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260812 11:16:27.372478 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260812 11:16:27.709393 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:16:28.269665 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260812 11:16:29.442885 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260812 11:16:29.794137 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260812 11:16:30.786376 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260812 11:16:31.383313 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:16:32.014707 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260812 11:16:32.381520 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260812 11:16:32.845830 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:16:33.225521 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260812 11:16:34.715550 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260812 11:16:35.130627 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260812 11:16:35.573021 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260812 11:16:36.746623 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:16:37.202764 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260812 11:16:37.729089 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260812 11:16:38.039813 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:16:38.040869 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260812 11:16:39.416088 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260812 11:16:40.014484 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260812 11:16:40.450033 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260812 11:16:42.020241 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:16:42.129372 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260812 11:16:42.291119 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260812 11:16:42.999284 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260812 11:16:43.414076 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20260812 11:16:44.770802 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260812 11:16:44.937472 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260812 11:16:45.585518 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260812 11:16:47.386698 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:16:47.388459 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260812 11:16:47.653563 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260812 11:16:47.766238 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260812 11:16:48.783684 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:16:49.858949 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260812 11:16:50.205420 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260812 11:16:50.242281 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260812 11:16:50.672049 17541 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (242 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:16:50.833056 17343 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (241 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:16:52.534261 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:16:52.535349 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260812 11:16:52.668689 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260812 11:16:52.709766 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.16.151.190:43065) | 127.16.151.190:43065 | UNAVAILABLE
Error from 127.16.151.190:43065: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.16.151.190:43065, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
FAILED
I20260812 11:16:53.602818 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.16.151.190:43065, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:16:53.794023 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:16:54.963140 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260812 11:16:55.093402 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260812 11:16:55.196736 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260812 11:16:57.539849 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:16:57.540798 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260812 11:16:57.559378 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260812 11:16:57.568327 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260812 11:16:59.271066 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:17:00.096383 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260812 11:17:00.234113 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260812 11:17:00.298519 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260812 11:17:02.383409 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260812 11:17:02.842716 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:17:02.919617 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260812 11:17:02.964303 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260812 11:17:04.364276 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:17:04.989886 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260812 11:17:05.490818 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260812 11:17:05.580556 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260812 11:17:07.586591 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260812 11:17:07.968051 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:17:08.163882 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260812 11:17:08.214488 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260812 11:17:09.473928 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:17:09.996374 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260812 11:17:10.602110 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260812 11:17:10.642220 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260812 11:17:12.226843 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260812 11:17:13.158396 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:17:13.159621 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260812 11:17:13.303406 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260812 11:17:14.745525 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260812 11:17:14.948863 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:17:16.014865 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260812 11:17:16.024377 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260812 11:17:17.646606 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260812 11:17:18.253595 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260812 11:17:18.469280 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260812 11:17:18.711730 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260812 11:17:20.269074 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260812 11:17:20.271765 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:17:20.912935 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260812 11:17:21.091938 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260812 11:17:22.770480 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260812 11:17:23.334314 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:17:23.335635 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260812 11:17:23.370167 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260812 11:17:25.342044 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260812 11:17:25.515571 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:17:25.967761 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260812 11:17:26.038853 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260812 11:17:27.788977 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260812 11:17:28.406313 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260812 11:17:28.442329 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:17:28.443318 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260812 11:17:30.277848 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260812 11:17:30.991539 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:17:30.992797 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260812 11:17:31.174912 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260812 11:17:33.064555 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260812 11:17:33.439885 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260812 11:17:33.590409 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260812 11:17:33.591544 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260812 11:17:35.700086 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260812 11:17:35.963749 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260812 11:17:35.971992 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260812 11:17:36.197511 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:17:38.154738 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260812 11:17:38.257398 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260812 11:17:38.760931 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:17:38.814093 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260812 11:17:40.436348 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260812 11:17:40.629645 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260812 11:17:41.285094 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:17:41.445410 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260812 11:17:42.627081 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260812 11:17:43.062153 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260812 11:17:43.975944 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:17:43.976861 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260812 11:17:45.362874 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260812 11:17:45.604504 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260812 11:17:46.326977 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:17:46.514964 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260812 11:17:47.961097 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260812 11:17:48.079282 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260812 11:17:48.759217 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260812 11:17:49.234519 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:17:50.352919 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260812 11:17:50.758576 17541 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (302 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:17:50.895183 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260812 11:17:50.923667 17343 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (301 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:17:50.933888 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260812 11:17:51.434206 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:17:53.050971 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260812 11:17:53.336495 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260812 11:17:53.364135 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260812 11:17:54.404855 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 21 similar messages]
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.16.151.190:43065) | 127.16.151.190:43065 | UNAVAILABLE
Error from 127.16.151.190:43065: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.16.151.190:43065, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
FAILED
I20260812 11:17:54.895092 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.16.151.190:43065, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:17:55.382512 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260812 11:17:55.638163 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260812 11:17:55.873513 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260812 11:17:56.442495 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:17:57.843219 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260812 11:17:57.994449 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260812 11:17:58.380743 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260812 11:17:59.480489 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:18:00.499181 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260812 11:18:00.647926 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260812 11:18:01.139644 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260812 11:18:01.454160 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:18:02.888445 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260812 11:18:03.309286 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260812 11:18:03.860926 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260812 11:18:04.710019 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:18:05.432744 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260812 11:18:05.712041 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260812 11:18:06.337718 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260812 11:18:07.046330 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:18:08.145802 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260812 11:18:08.155248 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260812 11:18:08.705487 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260812 11:18:09.772008 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:18:10.690001 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260812 11:18:10.757984 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260812 11:18:11.214761 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260812 11:18:12.067070 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:18:13.113093 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260812 11:18:13.554997 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260812 11:18:13.600956 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260812 11:18:14.989362 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:18:15.838523 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260812 11:18:15.916342 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260812 11:18:15.997471 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260812 11:18:17.399505 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:18:18.510018 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260812 11:18:18.511286 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260812 11:18:18.577060 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260812 11:18:20.006232 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:18:21.093889 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260812 11:18:21.101197 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260812 11:18:21.248031 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260812 11:18:22.751407 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:18:23.588546 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260812 11:18:23.670553 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260812 11:18:23.755776 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260812 11:18:25.252739 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:18:25.822868 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260812 11:18:26.230497 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260812 11:18:26.342928 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260812 11:18:27.843439 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:18:28.322068 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260812 11:18:28.648353 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260812 11:18:28.843617 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260812 11:18:30.430511 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:18:30.568336 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260812 11:18:30.989524 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260812 11:18:31.537201 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260812 11:18:32.908030 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:18:32.909565 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260812 11:18:33.449916 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260812 11:18:33.941279 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260812 11:18:35.493567 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260812 11:18:35.545778 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:18:35.890002 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260812 11:18:36.727946 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260812 11:18:38.240372 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:18:38.241827 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260812 11:18:38.517236 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260812 11:18:39.252446 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260812 11:18:40.796799 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260812 11:18:40.917912 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260812 11:18:41.290899 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260812 11:18:41.864248 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260812 11:18:42.918390 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260812 11:18:43.326918 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:18:43.673841 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260812 11:18:44.407642 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260812 11:18:45.342939 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260812 11:18:46.025645 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:18:46.082300 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260812 11:18:47.034350 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260812 11:18:47.628845 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260812 11:18:48.662014 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260812 11:18:48.749812 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:18:49.627664 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260812 11:18:50.170643 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260812 11:18:50.847910 17541 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (362 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:18:51.015507 17343 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (361 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:18:51.272411 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:18:51.273376 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260812 11:18:51.976792 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260812 11:18:52.785784 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260812 11:18:53.733064 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260812 11:18:53.942181 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:18:54.515843 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260812 11:18:55.598140 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.16.151.190:43065) | 127.16.151.190:43065 | UNAVAILABLE
Error from 127.16.151.190:43065: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.16.151.190:43065, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
FAILED
I20260812 11:18:56.175290 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.16.151.190:43065, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:18:56.225165 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260812 11:18:56.312078 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:18:56.885250 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260812 11:18:58.066828 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260812 11:18:58.591918 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260812 11:18:59.079981 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:18:59.224352 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260812 11:19:00.651087 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260812 11:19:01.261569 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260812 11:19:01.752444 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:19:01.753487 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260812 11:19:03.086385 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260812 11:19:03.945485 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260812 11:19:04.092903 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:19:04.275305 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260812 11:19:05.699599 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260812 11:19:06.470085 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260812 11:19:06.997972 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:19:06.998921 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260812 11:19:08.096839 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260812 11:19:09.147541 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260812 11:19:09.597167 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:19:09.623404 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260812 11:19:10.603376 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260812 11:19:11.604364 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260812 11:19:12.137784 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:19:12.278664 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260812 11:19:13.086336 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260812 11:19:13.990098 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260812 11:19:14.787369 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260812 11:19:14.947093 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:19:15.363113 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260812 11:19:16.330534 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260812 11:19:17.345559 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:19:17.466717 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260812 11:19:17.673098 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260812 11:19:18.883965 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260812 11:19:19.876312 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260812 11:19:20.251427 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:19:20.252857 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260812 11:19:21.156076 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260812 11:19:22.292119 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260812 11:19:22.682780 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260812 11:19:23.045753 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260812 11:19:23.694335 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260812 11:19:24.651558 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260812 11:19:25.352015 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:19:25.353091 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260812 11:19:26.327126 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260812 11:19:26.869613 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260812 11:19:27.796512 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260812 11:19:27.835466 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:19:28.797238 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260812 11:19:29.631083 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260812 11:19:30.392676 23743 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.ConnectToMaster -> {remote=127.16.151.190:43065, user_credentials={real_user=slave}} blocked reactor thread for 47594.5us
W20260812 11:19:30.593063 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:19:30.594863 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260812 11:19:31.476907 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260812 11:19:32.253113 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260812 11:19:32.852850 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260812 11:19:33.276311 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260812 11:19:33.886101 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260812 11:19:34.688457 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260812 11:19:36.024641 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:19:36.025624 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260812 11:19:36.391319 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260812 11:19:37.269836 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260812 11:19:38.130743 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260812 11:19:38.590538 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260812 11:19:38.598315 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260812 11:19:39.519471 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260812 11:19:41.157925 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:19:41.159021 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260812 11:19:41.171066 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260812 11:19:42.131664 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260812 11:19:43.325490 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260812 11:19:43.326582 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260812 11:19:43.722853 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260812 11:19:44.363134 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260812 11:19:45.885264 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260812 11:19:46.333187 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:19:46.334636 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260812 11:19:47.124799 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260812 11:19:48.519505 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:19:48.591090 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260812 11:19:48.707731 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260812 11:19:49.364856 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260812 11:19:50.934608 17541 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (422 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:19:51.104763 17343 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (421 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:19:51.207892 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260812 11:19:51.210114 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260812 11:19:51.655889 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:19:52.036166 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260812 11:19:53.548633 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260812 11:19:53.725021 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:19:53.726107 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260812 11:19:54.387048 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260812 11:19:56.185846 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260812 11:19:56.349890 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260812 11:19:57.149906 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260812 11:19:57.181396 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.16.151.190:43065) | 127.16.151.190:43065 | UNAVAILABLE
Error from 127.16.151.190:43065: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.16.151.190:43065, num_attempts: 343) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
FAILED
I20260812 11:19:57.250417 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.16.151.190:43065, num_attempts: 343) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:19:58.674865 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260812 11:19:58.677466 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260812 11:19:59.084316 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:19:59.675808 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260812 11:20:00.770627 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260812 11:20:00.933108 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260812 11:20:02.359133 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260812 11:20:02.520195 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:20:03.287217 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260812 11:20:03.657064 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260812 11:20:04.427187 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:20:04.826538 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260812 11:20:05.887687 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260812 11:20:06.079699 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260812 11:20:07.259433 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260812 11:20:08.029930 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:20:08.609620 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260812 11:20:08.612972 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260812 11:20:09.538847 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:20:09.967031 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260812 11:20:11.064508 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260812 11:20:11.132863 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260812 11:20:12.499615 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260812 11:20:13.135673 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:20:13.524700 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260812 11:20:13.722574 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260812 11:20:14.648334 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:20:14.974337 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260812 11:20:16.125871 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260812 11:20:16.551313 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260812 11:20:17.467093 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260812 11:20:18.385941 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:20:18.610817 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260812 11:20:18.763765 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260812 11:20:19.945883 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:20:19.946930 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260812 11:20:20.997442 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260812 11:20:21.422398 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260812 11:20:22.314795 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260812 11:20:23.592085 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260812 11:20:23.787767 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:20:23.788996 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260812 11:20:25.007352 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:20:25.008507 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260812 11:20:25.971259 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260812 11:20:26.403164 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260812 11:20:27.245234 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260812 11:20:28.311772 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260812 11:20:28.954298 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:20:28.955772 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260812 11:20:29.590612 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260812 11:20:30.210866 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260812 11:20:30.853003 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260812 11:20:31.520777 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260812 11:20:32.219926 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260812 11:20:33.191784 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260812 11:20:33.988473 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:20:33.989935 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260812 11:20:35.077311 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260812 11:20:35.470187 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:20:35.599162 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260812 11:20:36.663544 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260812 11:20:37.520817 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260812 11:20:37.840229 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260812 11:20:39.190171 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:20:39.191203 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260812 11:20:39.996393 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260812 11:20:40.577521 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:20:40.578701 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260812 11:20:41.731694 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260812 11:20:42.564640 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260812 11:20:43.013752 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260812 11:20:44.013407 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260812 11:20:44.604611 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:20:45.231282 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260812 11:20:45.516999 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260812 11:20:45.629455 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:20:46.390513 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260812 11:20:47.494951 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260812 11:20:48.000857 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260812 11:20:48.954706 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260812 11:20:49.714500 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260812 11:20:49.887953 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:20:50.371477 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260812 11:20:50.647454 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:20:51.020759 17541 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (482 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:20:51.200804 17343 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (481 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:20:51.424558 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260812 11:20:52.150790 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260812 11:20:52.649848 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260812 11:20:53.935354 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260812 11:20:54.663033 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260812 11:20:55.069110 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260812 11:20:55.284847 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:20:55.650672 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:20:56.340062 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260812 11:20:57.019899 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260812 11:20:57.581609 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.16.151.190:43065) | 127.16.151.190:43065 | UNAVAILABLE
Error from 127.16.151.190:43065: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.16.151.190:43065, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
FAILED
I20260812 11:20:58.548406 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.16.151.190:43065, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:20:58.647500 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260812 11:20:59.476123 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260812 11:21:00.095822 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260812 11:21:00.562297 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:21:00.960229 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260812 11:21:00.996652 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260812 11:21:01.962677 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260812 11:21:02.754527 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260812 11:21:03.737658 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260812 11:21:04.691556 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260812 11:21:05.424477 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260812 11:21:05.911355 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:21:06.130359 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:21:06.532673 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260812 11:21:07.078811 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260812 11:21:07.822256 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260812 11:21:08.999218 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260812 11:21:09.677528 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260812 11:21:10.361667 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260812 11:21:11.206334 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:21:11.410475 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:21:11.411722 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260812 11:21:12.287954 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260812 11:21:13.065534 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260812 11:21:13.811043 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260812 11:21:14.714535 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260812 11:21:15.371230 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260812 11:21:16.267931 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260812 11:21:16.305642 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:21:16.860762 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:21:17.286705 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260812 11:21:17.672039 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260812 11:21:18.763087 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260812 11:21:19.724186 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260812 11:21:20.168323 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260812 11:21:21.314798 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:21:21.511292 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260812 11:21:21.962736 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:21:22.408150 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260812 11:21:22.701283 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260812 11:21:24.128063 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260812 11:21:24.725914 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260812 11:21:25.295074 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260812 11:21:26.613584 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:21:26.683907 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260812 11:21:27.001840 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260812 11:21:27.146543 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:21:27.570863 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260812 11:21:29.007148 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260812 11:21:29.779656 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260812 11:21:30.033293 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260812 11:21:31.349416 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260812 11:21:31.967927 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:21:32.334846 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:21:32.543737 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260812 11:21:32.599663 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260812 11:21:33.954411 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260812 11:21:35.121806 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260812 11:21:35.464318 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260812 11:21:36.550226 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260812 11:21:37.020540 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260812 11:21:37.551808 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260812 11:21:37.731381 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:21:38.070660 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260812 11:21:39.332599 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260812 11:21:40.052721 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260812 11:21:40.501215 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260812 11:21:42.096508 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260812 11:21:42.136605 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:21:42.637893 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260812 11:21:42.877048 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260812 11:21:43.070787 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:21:44.454779 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260812 11:21:45.255395 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260812 11:21:45.486738 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260812 11:21:47.187770 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260812 11:21:47.386637 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:21:47.652331 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260812 11:21:47.928169 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260812 11:21:48.554226 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:21:49.456859 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260812 11:21:50.220253 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260812 11:21:50.552814 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260812 11:21:51.107903 17541 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (542 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:21:51.288959 17343 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (541 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:21:51.943958 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260812 11:21:52.510675 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260812 11:21:53.048202 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260812 11:21:53.380097 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260812 11:21:53.639833 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:21:54.515394 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260812 11:21:55.457283 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260812 11:21:55.794147 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260812 11:21:57.225407 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260812 11:21:57.774887 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:21:57.905498 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260812 11:21:58.359711 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260812 11:21:59.030016 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:21:59.618418 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.16.151.190:43065) | 127.16.151.190:43065 | UNAVAILABLE
Error from 127.16.151.190:43065: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.16.151.190:43065, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
FAILED
I20260812 11:21:59.859254 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.16.151.190:43065, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:22:00.436529 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260812 11:22:00.751087 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260812 11:22:02.117086 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260812 11:22:02.897611 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:22:02.933506 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260812 11:22:03.393700 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260812 11:22:04.466156 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:22:04.467602 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260812 11:22:05.580430 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260812 11:22:05.717715 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260812 11:22:07.045653 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260812 11:22:08.103441 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:22:08.104522 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260812 11:22:08.371608 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260812 11:22:09.646373 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:22:09.647621 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260812 11:22:10.545540 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260812 11:22:10.894172 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260812 11:22:12.130511 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260812 11:22:12.890216 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260812 11:22:13.351679 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:22:13.352723 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260812 11:22:14.636555 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260812 11:22:15.182314 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:22:15.288043 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260812 11:22:15.944869 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260812 11:22:17.071988 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260812 11:22:17.799291 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260812 11:22:18.210722 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260812 11:22:18.611369 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260812 11:22:19.765928 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260812 11:22:20.307685 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260812 11:22:20.361217 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:22:20.673681 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260812 11:22:22.232496 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260812 11:22:22.737191 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260812 11:22:23.149387 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260812 11:22:23.686364 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:22:25.001327 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260812 11:22:25.361284 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260812 11:22:25.411342 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:22:25.644861 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260812 11:22:27.503839 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260812 11:22:27.813329 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260812 11:22:28.048702 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260812 11:22:28.974241 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:22:29.901489 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260812 11:22:30.471052 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260812 11:22:30.555677 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260812 11:22:30.815455 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:22:32.505287 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260812 11:22:33.103895 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260812 11:22:33.151520 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260812 11:22:34.060840 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:22:34.753556 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260812 11:22:35.539872 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260812 11:22:35.723357 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260812 11:22:35.842124 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:22:37.489490 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260812 11:22:38.052419 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260812 11:22:38.182741 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260812 11:22:39.065512 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:22:40.103386 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260812 11:22:40.619810 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260812 11:22:40.804929 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260812 11:22:40.955837 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:22:42.314525 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260812 11:22:42.962656 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260812 11:22:43.133881 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260812 11:22:44.130698 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:22:44.836318 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260812 11:22:45.402694 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260812 11:22:45.403651 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260812 11:22:46.520520 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:22:47.683665 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260812 11:22:48.037773 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260812 11:22:48.149727 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260812 11:22:49.174096 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260812 11:22:50.508587 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260812 11:22:50.672183 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260812 11:22:50.802284 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260812 11:22:51.192510 17541 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (602 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:22:51.376904 17343 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (601 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:22:51.618898 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20260812 11:22:52.726617 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260812 11:22:53.251039 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260812 11:22:53.307788 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260812 11:22:54.187958 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260812 11:22:55.173844 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260812 11:22:55.538666 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260812 11:22:55.723829 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260812 11:22:56.977514 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:22:57.653234 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260812 11:22:57.876283 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260812 11:22:57.940001 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260812 11:22:59.494197 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:23:00.032418 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260812 11:23:00.078275 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260812 11:23:00.614806 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.16.151.190:43065) | 127.16.151.190:43065 | UNAVAILABLE
Error from 127.16.151.190:43065: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.16.151.190:43065, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
FAILED
I20260812 11:23:01.203464 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.16.151.190:43065, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:23:02.360281 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20260812 11:23:02.361680 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260812 11:23:02.745442 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260812 11:23:03.069105 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260812 11:23:04.661091 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:23:04.950878 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260812 11:23:05.452214 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260812 11:23:05.653877 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260812 11:23:07.259320 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260812 11:23:07.640276 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:23:07.646651 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260812 11:23:08.261484 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260812 11:23:09.441220 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260812 11:23:10.044310 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260812 11:23:10.156136 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260812 11:23:10.462174 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260812 11:23:12.087783 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260812 11:23:12.616025 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260812 11:23:12.866053 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260812 11:23:12.965046 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:23:14.552692 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260812 11:23:15.006812 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260812 11:23:15.298112 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260812 11:23:15.299137 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260812 11:23:17.134397 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260812 11:23:17.600109 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260812 11:23:17.634125 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260812 11:23:18.221954 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:23:19.693427 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260812 11:23:20.045908 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260812 11:23:20.491372 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:23:20.492386 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260812 11:23:22.229601 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260812 11:23:22.628618 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260812 11:23:23.052716 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260812 11:23:23.329717 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:23:24.765811 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260812 11:23:25.239789 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260812 11:23:25.356721 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260812 11:23:25.806223 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:23:27.498395 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260812 11:23:27.755062 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260812 11:23:28.225368 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260812 11:23:28.489480 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:23:30.229938 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260812 11:23:30.279947 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260812 11:23:30.853766 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260812 11:23:31.026898 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260812 11:23:32.587275 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260812 11:23:32.759194 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260812 11:23:33.559806 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:23:33.623368 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260812 11:23:35.058578 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260812 11:23:35.264549 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260812 11:23:36.071384 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:23:36.072403 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260812 11:23:37.340759 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260812 11:23:37.511560 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260812 11:23:38.526947 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260812 11:23:38.693888 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:23:39.566481 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260812 11:23:39.584178 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260812 11:23:41.024040 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260812 11:23:41.126710 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260812 11:23:42.080459 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260812 11:23:42.101228 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260812 11:23:43.800420 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260812 11:23:44.069262 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:23:44.650548 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260812 11:23:44.702626 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260812 11:23:45.991897 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260812 11:23:46.306149 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:23:47.083735 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260812 11:23:47.357321 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260812 11:23:48.605906 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260812 11:23:49.370895 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:23:49.827612 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260812 11:23:49.877449 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260812 11:23:51.227330 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260812 11:23:51.280340 17541 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (662 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:23:51.465230 17343 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (661 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:23:51.487636 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:23:52.180783 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260812 11:23:52.607952 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260812 11:23:53.692775 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260812 11:23:54.634150 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:23:54.635608 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260812 11:23:55.166584 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260812 11:23:55.824258 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260812 11:23:56.763130 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:23:57.046770 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260812 11:23:57.959933 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260812 11:23:58.491319 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260812 11:23:59.630084 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260812 11:24:00.185043 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:24:00.329732 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260812 11:24:00.868355 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260812 11:24:02.020385 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:24:02.092574 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.16.151.190:43065) | 127.16.151.190:43065 | UNAVAILABLE
Error from 127.16.151.190:43065: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.16.151.190:43065, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
FAILED
I20260812 11:24:02.548676 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.16.151.190:43065, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:24:02.721092 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260812 11:24:03.243723 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260812 11:24:04.896028 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260812 11:24:05.353973 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:24:05.375074 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260812 11:24:05.802314 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260812 11:24:07.354082 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:24:07.403784 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260812 11:24:07.961876 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260812 11:24:08.577889 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260812 11:24:09.985124 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260812 11:24:10.284142 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260812 11:24:10.567045 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:24:11.070884 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260812 11:24:12.541704 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:24:12.752877 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260812 11:24:12.917291 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260812 11:24:13.675082 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260812 11:24:14.938133 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260812 11:24:15.166347 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260812 11:24:16.033864 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260812 11:24:16.048579 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:24:17.542904 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260812 11:24:17.554766 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:24:17.555876 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260812 11:24:18.621935 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260812 11:24:20.198012 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260812 11:24:20.244144 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260812 11:24:20.915162 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260812 11:24:21.298801 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:24:22.606619 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:24:22.607620 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260812 11:24:22.834017 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260812 11:24:23.483858 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260812 11:24:25.153762 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260812 11:24:25.579396 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260812 11:24:26.062613 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260812 11:24:26.379455 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:24:27.783918 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260812 11:24:27.904026 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:24:27.905086 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260812 11:24:28.588980 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260812 11:24:30.580761 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260812 11:24:30.665078 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260812 11:24:31.207994 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260812 11:24:31.467746 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:24:32.919795 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260812 11:24:33.212782 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:24:33.312275 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260812 11:24:33.740259 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260812 11:24:35.477809 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260812 11:24:35.992273 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260812 11:24:36.023699 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260812 11:24:37.017975 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:24:37.948580 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260812 11:24:38.497014 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:24:38.498122 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260812 11:24:38.598623 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260812 11:24:40.460995 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260812 11:24:41.221818 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260812 11:24:41.290597 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260812 11:24:42.497543 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:24:43.075088 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260812 11:24:43.755923 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:24:43.756958 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260812 11:24:43.870476 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260812 11:24:45.653880 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260812 11:24:46.359867 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260812 11:24:46.497534 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260812 11:24:47.567449 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:24:48.044059 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260812 11:24:48.482858 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260812 11:24:48.917838 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:24:48.918825 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260812 11:24:50.206385 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260812 11:24:51.174788 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260812 11:24:51.369226 17541 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (722 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:24:51.501958 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260812 11:24:51.555294 17343 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (721 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:24:52.767593 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:24:52.768643 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260812 11:24:53.711226 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260812 11:24:54.122483 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:24:54.185000 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260812 11:24:55.461722 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260812 11:24:56.021766 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260812 11:24:56.792430 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260812 11:24:57.954782 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:24:57.955782 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260812 11:24:58.672528 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260812 11:24:59.081202 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260812 11:24:59.176407 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:25:00.437762 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260812 11:25:01.381098 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260812 11:25:01.805677 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260812 11:25:02.882377 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260812 11:25:03.386619 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:25:03.805421 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.16.151.190:43065) | 127.16.151.190:43065 | UNAVAILABLE
Error from 127.16.151.190:43065: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.16.151.190:43065, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
FAILED
I20260812 11:25:03.879051 16990 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.16.151.190:43065, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
W20260812 11:25:04.301122 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260812 11:25:04.431819 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260812 11:25:05.377266 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260812 11:25:06.337636 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260812 11:25:07.046070 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260812 11:25:08.008697 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260812 11:25:08.418336 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:25:08.781404 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260812 11:25:09.350487 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:25:09.408442 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260812 11:25:10.512709 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260812 11:25:11.266454 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260812 11:25:11.922887 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260812 11:25:13.016508 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260812 11:25:13.476779 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260812 11:25:13.477254 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:25:14.447870 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:25:14.868688 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260812 11:25:15.591984 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260812 11:25:15.997728 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260812 11:25:17.487383 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260812 11:25:18.160822 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
************************ BEGIN STACKS **************************
W20260812 11:25:18.610327 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260812 11:25:18.641804 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
[New LWP 16993]
[New LWP 16995]
[New LWP 16998]
[New LWP 17741]
[New LWP 17742]
[New LWP 17744]
[New LWP 17748]
[New LWP 24922]
[New LWP 24923]
[New LWP 24925]
[New LWP 24926]
[New LWP 24929]
[New LWP 24930]
[New LWP 24931]
[New LWP 24932]
[New LWP 24934]
W20260812 11:25:19.462978 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:25:19.987955 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260812 11:25:20.734901 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260812 11:25:20.925594 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260812 11:25:22.583007 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260812 11:25:23.041299 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260812 11:25:23.262674 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260812 11:25:23.655506 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:25:24.638871 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:25:25.293262 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260812 11:25:25.499938 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260812 11:25:25.838881 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260812 11:25:27.948264 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260812 11:25:28.023069 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260812 11:25:28.145015 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260812 11:25:29.181674 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:25:29.717916 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:25:30.302302 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260812 11:25:30.469264 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260812 11:25:30.883548 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260812 11:25:32.872079 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260812 11:25:33.044164 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260812 11:25:33.452211 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260812 11:25:34.315346 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:25:34.790724 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:25:35.288779 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260812 11:25:35.486460 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260812 11:25:36.056457 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260812 11:25:37.830111 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260812 11:25:38.028363 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260812 11:25:38.393226 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260812 11:25:39.475564 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260812 11:25:39.883361 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260812 11:25:40.300240 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260812 11:25:40.494922 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260812 11:25:41.060564 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
W20260812 11:25:42.784706 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260812 11:25:42.976442 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260812 11:25:43.461524 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260812 11:25:44.841404 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260812 11:25:45.076779 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260812 11:25:45.396510 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
W20260812 11:25:45.562091 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
0x00007fe3d0f95ad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
W20260812 11:25:46.283736 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
Id Target Id Frame
* 1 Thread 0x7fe3c9496d40 (LWP 16990) "all_types-itest" 0x00007fe3d0f95ad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
2 Thread 0x7fe3c4fbd700 (LWP 16993) "all_types-itest" 0x00007fe3d0f9a32a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
3 Thread 0x7fe3c39f4700 (LWP 16995) "kernel-watcher-" 0x00007fe3d0f95fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
4 Thread 0x7fe3c31f3700 (LWP 16998) "rpc reactor-169" 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
5 Thread 0x7fe3c29f2700 (LWP 17741) "rpc reactor-177" 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
6 Thread 0x7fe3c1fea700 (LWP 17742) "rpc reactor-177" 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
7 Thread 0x7fe3c17e9700 (LWP 17744) "rpc reactor-177" 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
8 Thread 0x7fe3c0fe8700 (LWP 17748) "rpc reactor-177" 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
9 Thread 0x7fe3bc565700 (LWP 24922) "rpc reactor-249" 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
10 Thread 0x7fe3bad26700 (LWP 24923) "rpc reactor-249" 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
11 Thread 0x7fe3bb53d700 (LWP 24925) "rpc reactor-249" 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
12 Thread 0x7fe3be8b1700 (LWP 24926) "rpc reactor-249" 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
13 Thread 0x7fe3bf0b2700 (LWP 24929) "rpc reactor-249" 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
14 Thread 0x7fe3be0b0700 (LWP 24930) "rpc reactor-249" 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
15 Thread 0x7fe3bd593700 (LWP 24931) "rpc reactor-249" 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
16 Thread 0x7fe3bcd82700 (LWP 24932) "rpc reactor-249" 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
17 Thread 0x7fe3bbd54700 (LWP 24934) "client-negotiat" 0x00007fe3d0f95fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
Thread 17 (Thread 0x7fe3bbd54700 (LWP 24934)):
#0 0x00007fe3d0f95fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fe3ce1808d9 in kudu::ConditionVariable::WaitFor (this=0x7fe3d0f95fb9 <pthread_cond_timedwait@@GLIBC_2.3.2+649>, delta=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/condition_variable.cc:121
#2 0x00007fe3ce430e0a in kudu::ThreadPool::DispatchThread (this=0x616000005480) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:721
#3 0x00007fe3ce43491c in kudu::ThreadPool::CreateThread()::$_2::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:849
#4 0x00007fe3ce434799 in std::_Function_handler<void (), kudu::ThreadPool::CreateThread()::$_2>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#5 0x00007fe3e298bbbd in std::function<void ()>::operator()() const (this=0x611000137d28) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#6 0x00007fe3ce40feaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#7 0x00007fe3d0f8f6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#8 0x00007fe3ca26961f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 16 (Thread 0x7fe3bcd82700 (LWP 24932)):
#0 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fe3cc358bda in epoll_poll (loop=0x61900020b280, timeout=0.099352071061730385) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fe3cc35c228 in ev_run (loop=0x61900020b280, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fe3d49f4e31 in ev::loop_ref::run (this=this@entry=0x614000002888, flags=7073152, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fe3d49ed326 in kudu::rpc::ReactorThread::RunThread (this=0x614000002880) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007fe3d49f23ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007fe3d49f2229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007fe3e298bbbd in std::function<void ()>::operator()() const (this=0x6110000abd28) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007fe3ce40feaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007fe3d0f8f6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fe3ca26961f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 15 (Thread 0x7fe3bd593700 (LWP 24931)):
#0 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fe3cc358bda in epoll_poll (loop=0x61900020b780, timeout=0.0993532610591501) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fe3cc35c228 in ev_run (loop=0x61900020b780, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fe3d49f4e31 in ev::loop_ref::run (this=this@entry=0x614000001e88, flags=7072256, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fe3d49ed326 in kudu::rpc::ReactorThread::RunThread (this=0x614000001e80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007fe3d49f23ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007fe3d49f2229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007fe3e298bbbd in std::function<void ()>::operator()() const (this=0x611000118a68) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007fe3ce40feaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007fe3d0f8f6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fe3ca26961f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 14 (Thread 0x7fe3be0b0700 (LWP 24930)):
#0 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fe3cc358bda in epoll_poll (loop=0x61900020bc80, timeout=0.099340637214481831) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fe3cc35c228 in ev_run (loop=0x61900020bc80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fe3d49f4e31 in ev::loop_ref::run (this=this@entry=0x614000001a88, flags=7071360, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fe3d49ed326 in kudu::rpc::ReactorThread::RunThread (this=0x614000001a80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007fe3d49f23ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007fe3d49f2229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007fe3e298bbbd in std::function<void ()>::operator()() const (this=0x611000118e28) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007fe3ce40feaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007fe3d0f8f6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fe3ca26961f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 13 (Thread 0x7fe3bf0b2700 (LWP 24929)):
#0 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fe3cc358bda in epoll_poll (loop=0x61900020c180, timeout=0.099355221260339022) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fe3cc35c228 in ev_run (loop=0x61900020c180, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fe3d49f4e31 in ev::loop_ref::run (this=this@entry=0x614000001c88, flags=7142144, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fe3d49ed326 in kudu::rpc::ReactorThread::RunThread (this=0x614000001c80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007fe3d49f23ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007fe3d49f2229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007fe3e298bbbd in std::function<void ()>::operator()() const (this=0x6110001190a8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007fe3ce40feaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007fe3d0f8f6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fe3ca26961f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 12 (Thread 0x7fe3be8b1700 (LWP 24926)):
#0 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fe3cc358bda in epoll_poll (loop=0x61900020f380, timeout=0.09917925694026053) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fe3cc35c228 in ev_run (loop=0x61900020f380, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fe3d49f4e31 in ev::loop_ref::run (this=this@entry=0x614000001888, flags=7139456, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fe3d49ed326 in kudu::rpc::ReactorThread::RunThread (this=0x614000001880) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007fe3d49f23ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007fe3d49f2229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007fe3e298bbbd in std::function<void ()>::operator()() const (this=0x611000130668) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007fe3ce40feaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007fe3d0f8f6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fe3ca26961f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 11 (Thread 0x7fe3bb53d700 (LWP 24925)):
#0 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fe3cc358bda in epoll_poll (loop=0x61900020f880, timeout=0.099146737018600106) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fe3cc35c228 in ev_run (loop=0x61900020f880, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fe3d49f4e31 in ev::loop_ref::run (this=this@entry=0x614000002088, flags=7138560, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fe3d49ed326 in kudu::rpc::ReactorThread::RunThread (this=0x614000002080) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007fe3d49f23ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007fe3d49f2229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007fe3e298bbbd in std::function<void ()>::operator()() const (this=0x6110001308e8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007fe3ce40feaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007fe3d0f8f6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fe3ca26961f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 10 (Thread 0x7fe3bad26700 (LWP 24923)):
#0 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fe3cc358bda in epoll_poll (loop=0x6190001d4c80, timeout=0.099154917057603598) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fe3cc35c228 in ev_run (loop=0x6190001d4c80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fe3d49f4e31 in ev::loop_ref::run (this=this@entry=0x614000020288, flags=7137664, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fe3d49ed326 in kudu::rpc::ReactorThread::RunThread (this=0x614000020280) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007fe3d49f23ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007fe3d49f2229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007fe3e298bbbd in std::function<void ()>::operator()() const (this=0x611000130528) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007fe3ce40feaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007fe3d0f8f6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fe3ca26961f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 9 (Thread 0x7fe3bc565700 (LWP 24922)):
#0 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fe3cc358bda in epoll_poll (loop=0x6190001d5180, timeout=0.099088436225429177) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fe3cc35c228 in ev_run (loop=0x6190001d5180, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fe3d49f4e31 in ev::loop_ref::run (this=this@entry=0x614000020088, flags=7136768, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fe3d49ed326 in kudu::rpc::ReactorThread::RunThread (this=0x614000020080) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007fe3d49f23ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007fe3d49f2229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007fe3e298bbbd in std::function<void ()>::operator()() const (this=0x611000130a28) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007fe3ce40feaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007fe3d0f8f6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fe3ca26961f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 8 (Thread 0x7fe3c0fe8700 (LWP 17748)):
#0 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fe3cc358bda in epoll_poll (loop=0x619000055a80, timeout=0.099139714380726218) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fe3cc35c228 in ev_run (loop=0x619000055a80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fe3d49f4e31 in ev::loop_ref::run (this=this@entry=0x614000004288, flags=16256, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fe3d49ed326 in kudu::rpc::ReactorThread::RunThread (this=0x614000004280) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007fe3d49f23ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007fe3d49f2229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007fe3e298bbbd in std::function<void ()>::operator()() const (this=0x6110000589e8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007fe3ce40feaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007fe3d0f8f6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fe3ca26961f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 7 (Thread 0x7fe3c17e9700 (LWP 17744)):
#0 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fe3cc358bda in epoll_poll (loop=0x619000055580, timeout=0.09925908176228404) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fe3cc35c228 in ev_run (loop=0x619000055580, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fe3d49f4e31 in ev::loop_ref::run (this=this@entry=0x614000004088, flags=15360, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fe3d49ed326 in kudu::rpc::ReactorThread::RunThread (this=0x614000004080) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007fe3d49f23ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007fe3d49f2229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007fe3e298bbbd in std::function<void ()>::operator()() const (this=0x6110000588a8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007fe3ce40feaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007fe3d0f8f6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fe3ca26961f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 6 (Thread 0x7fe3c1fea700 (LWP 17742)):
#0 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fe3cc358bda in epoll_poll (loop=0x619000055080, timeout=0.099032712401822209) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fe3cc35c228 in ev_run (loop=0x619000055080, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fe3d49f4e31 in ev::loop_ref::run (this=this@entry=0x614000003e88, flags=14464, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fe3d49ed326 in kudu::rpc::ReactorThread::RunThread (this=0x614000003e80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007fe3d49f23ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007fe3d49f2229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007fe3e298bbbd in std::function<void ()>::operator()() const (this=0x611000058768) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007fe3ce40feaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007fe3d0f8f6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fe3ca26961f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 5 (Thread 0x7fe3c29f2700 (LWP 17741)):
#0 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fe3cc358bda in epoll_poll (loop=0x619000054b80, timeout=0.098985204473137856) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fe3cc35c228 in ev_run (loop=0x619000054b80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fe3d49f4e31 in ev::loop_ref::run (this=this@entry=0x614000003c88, flags=13568, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fe3d49ed326 in kudu::rpc::ReactorThread::RunThread (this=0x614000003c80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007fe3d49f23ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007fe3d49f2229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007fe3e298bbbd in std::function<void ()>::operator()() const (this=0x611000058628) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007fe3ce40feaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007fe3d0f8f6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fe3ca26961f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 4 (Thread 0x7fe3c31f3700 (LWP 16998)):
#0 0x00007fe3ca269947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fe3cc358bda in epoll_poll (loop=0x619000052880, timeout=0.09912397712469101) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fe3cc35c228 in ev_run (loop=0x619000052880, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fe3d49f4e31 in ev::loop_ref::run (this=this@entry=0x614000003488, flags=9088, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fe3d49ed326 in kudu::rpc::ReactorThread::RunThread (this=0x614000003480) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007fe3d49f23ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007fe3d49f2229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007fe3e298bbbd in std::function<void ()>::operator()() const (this=0x611000051fa8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007fe3ce40feaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007fe3d0f8f6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fe3ca26961f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 3 (Thread 0x7fe3c39f4700 (LWP 16995)):
#0 0x00007fe3d0f95fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fe3ce18050e in kudu::ConditionVariable::WaitUntil (this=0x612000009118, until=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/condition_variable.cc:87
W20260812 11:25:47.870374 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
#2 0x00007fe3e1fd302c in kudu::CountDownLatch::WaitUntil (this=<optimized out>, when=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/countdown_latch.h:89
#3 0x00007fe3e1fcb4aa in kudu::CountDownLatch::WaitFor (this=0x612000009140, delta=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/countdown_latch.h:99
#4 0x00007fe3ce2b645f in kudu::KernelStackWatchdog::RunThread (this=0x612000009040) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/kernel_stack_watchdog.cc:134
#5 0x00007fe3ce2b751c in kudu::KernelStackWatchdog::KernelStackWatchdog()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/kernel_stack_watchdog.cc:71
#6 0x00007fe3ce2b7399 in std::_Function_handler<void (), kudu::KernelStackWatchdog::KernelStackWatchdog()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007fe3e298bbbd in std::function<void ()>::operator()() const (this=0x611000052228) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007fe3ce40feaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007fe3d0f8f6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fe3ca26961f in clone () from /lib/x86_64-linux-gnu/libc.so.6
W20260812 11:25:48.442822 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
Thread 2 (Thread 0x7fe3c4fbd700 (LWP 16993)):
#0 0x00007fe3d0f9a32a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000039b4fb in __interceptor_waitpid (pid=<optimized out>, status=0x7fe3c4fbb8b0, options=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2434
#2 0x00007fe3ce3fa796 in kudu::Subprocess::DoWait (this=<optimized out>, wait_status=<optimized out>, mode=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/subprocess.cc:817
#3 0x00007fe3ce3fa4c5 in kudu::Subprocess::Wait (this=0x7fe3c4fbb8b0, wait_status=0x0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/subprocess.cc:544
#4 0x00007fe3ce2902be in kudu::PstackWatcher::RunStackDump (argv=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:258
#5 0x00007fe3ce28f9a7 in kudu::PstackWatcher::RunGdbStackDump (pid=<optimized out>, flags=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:218
#6 0x00007fe3ce28ee96 in kudu::PstackWatcher::DumpPidStacks (pid=<optimized out>, flags=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:169
#7 0x00007fe3ce28d417 in kudu::PstackWatcher::DumpStacks (flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:162
#8 0x00000000005386f9 in kudu::CreateAndStartTimeoutThread()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:60
#9 0x0000000000538583 in _ZSt13__invoke_implIvZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0JEET_St14__invoke_otherOT0_DpOT1_ (__f=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:60
#10 0x0000000000538503 in _ZSt8__invokeIZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS3_DpOS4_ (__fn=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:95
#11 0x00000000005384d9 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> >::_M_invoke<0ul> (this=this@entry=0x602000029c18) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:244
#12 0x0000000000538499 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> >::operator()() (this=this@entry=0x602000029c18) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:253
#13 0x000000000053839b in std::thread::_State_impl<std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> > >::_M_run() (this=<optimized out>) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:196
#14 0x00007fe3d58d6a4f in execute_native_thread_routine () from /tmp/dist-test-tasknusDu0/build/asan/bin/../../../thirdparty/installed/uninstrumented/lib/librocksdb.so.7
#15 0x00007fe3d0f8f6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#16 0x00007fe3ca26961f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 1 (Thread 0x7fe3c9496d40 (LWP 16990)):
#0 0x00007fe3d0f95ad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fe3ce1801e7 in kudu::ConditionVariable::Wait (this=0x60e0001b55f0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/condition_variable.cc:55
#2 0x00007fe3e30421b9 in kudu::CountDownLatch::Wait (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/countdown_latch.h:79
W20260812 11:25:48.887126 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
#3 0x00007fe3e2738f4b in kudu::Synchronizer::Wait (this=0x7ffefd667420) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/async_util.h:58
#4 0x00007fe3db9d9ef3 in kudu::client::KuduClient::Data::ConnectToCluster (this=0x615000c2dd80, client=<optimized out>, deadline=..., creds_policy=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/client/client-internal.cc:815
W20260812 11:25:49.990159 17170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 9 similar messages]
#5 0x00007fe3db959901 in kudu::client::KuduClientBuilder::Build (this=0x7ffefd667d20, client=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/client/client.cc:469
W20260812 11:25:50.420987 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260812 11:25:50.581964 17170 consensus_peers.cc:487] T 6dda8ac5b4254f1296f26326220247e2 P b7094e9660dc47ceb5e9f70e089ef166 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
#6 0x00007fe3dc2ffeac in kudu::tools::CreateKuduClient (master_addresses=..., client=<optimized out>, can_see_all_replicas=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/tool_action_common.cc:738
W20260812 11:25:51.071815 17371 consensus_peers.cc:487] T 88f0bbdf7cac496198e6b83299d50850 P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
W20260812 11:25:51.399821 17371 consensus_peers.cc:487] T a99dd1ffecd04818b07c06d01f830f1a P 3884e2014802405c8f38db8f6dcabc25 -> Peer 5438c5a3070f40d99bb7955dc412fbcc (127.16.151.131:38427): Couldn't send request to peer 5438c5a3070f40d99bb7955dc412fbcc. Status: Network error: Client connection negotiation failed: client connection to 127.16.151.131:38427: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
W20260812 11:25:51.463002 17541 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (782 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
#7 0x00007fe3e1ff8c15 in kudu::tools::RemoteKsckCluster::Connect (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck_remote.cc:528
#8 0x00007fe3e1f6c87e in kudu::tools::Ksck::CheckClusterRunning (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck.cc:460
#9 0x00007fe3e1f6e83f in kudu::tools::Ksck::Run (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck.cc:595
#10 0x00007fe3e1f7313d in kudu::tools::Ksck::RunAndPrintResults (this=0x61900020c690) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck.cc:801
W20260812 11:25:51.649714 17343 heartbeater.cc:644] Failed to heartbeat to 127.16.151.190:43065 (781 consecutive failures): Network error: Failed to ping master at 127.16.151.190:43065: Client connection negotiation failed: client connection to 127.16.151.190:43065: connect: Connection refused (error 111)
#11 0x00007fe3e2feadd3 in kudu::ClusterVerifier::RunKsck (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/cluster_verifier.cc:113
#12 0x00007fe3e2fea556 in kudu::ClusterVerifier::CheckCluster (this=0x7ffefd669140) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/cluster_verifier.cc:78
#13 0x0000000000461258 in kudu::client::AllTypesItest<kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)3> > >::RunTest(std::function<kudu::Status (kudu::client::KuduScanner*)> const&, std::function<void (kudu::client::KuduScanBatch const&, int, int*)> const&) (this=0x60c00000d000, scanner_setup=..., verifier=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/all_types-itest.cc:622
#14 0x00000000005096cd in kudu::client::AllTypesItest_TestTimestampPadding_Test<kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)3> > >::TestBody (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/all_types-itest.cc:804
#15 0x00007fe3cec69bfe in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (location=0x7fe3cec6fa28 "the test body", method=<optimized out>, object=0x60c00000d000) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest-internal-inl.h:831
#16 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0x60c00000d000, method=<optimized out>, location=location@entry=0x7fe3cec6fa28 "the test body") at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#17 0x00007fe3cec5e5db in testing::Test::Run (this=0x60c00000d000) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2674
#18 testing::Test::Run (this=0x60c00000d000) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2664
#19 0x00007fe3cec5e795 in testing::TestInfo::Run (this=0x611000047080) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853
#20 0x00007fe3cec5ecdf in testing::TestSuite::Run (this=0x611000045780) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012
#21 testing::TestSuite::Run (this=0x611000045780) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2985
#22 0x00007fe3cec5f29f in testing::internal::UnitTestImpl::RunAllTests() () at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest-internal-inl.h:600
#23 0x00007fe3cec6a13e in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x7fe3cec715a8 "auxiliary test code (environments or event listeners)", method=<optimized out>, object=0x61600000b480) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest-internal-inl.h:831
#24 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x61600000b480, method=<optimized out>, location=location@entry=0x7fe3cec715a8 "auxiliary test code (environments or event listeners)") at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#25 0x00007fe3cec5e83d in testing::UnitTest::Run() () at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/include/gtest/gtest.h:1246
#26 0x0000000000539482 in RUN_ALL_TESTS () at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/gtest/gtest.h:2293
#27 0x0000000000537992 in main (argc=<optimized out>, argv=0x7ffefd6699c8) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:109
************************* END STACKS ***************************
F20260812 11:25:52.680630 16993 test_main.cc:63] Maximum unit test time exceeded (870 sec)
*** Check failure stack trace: ***
*** Aborted at 1786533952 (unix time) try "date -d @1786533952" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e80000425e) received by PID 16990 (TID 0x7fe3c4fbd700) from PID 16990; stack trace: ***
@ 0x7fe3cce58a46 google::(anonymous namespace)::FailureSignalHandler() at ??:0
@ 0x7fe3d0f9a980 (unknown) at ??:0
@ 0x7fe3ca186e87 gsignal at ??:0
@ 0x7fe3ca1887f1 abort at ??:0
@ 0x7fe3cce4a9ad google::LogMessage::Fail() at ??:0
@ 0x7fe3cce4d88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe3cce4a4de google::LogMessage::Flush() at ??:0
@ 0x7fe3cce4ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x538891 kudu::CreateAndStartTimeoutThread()::$_0::operator()() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:63
@ 0x538583 std::__invoke_impl<>() at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:60
@ 0x538503 std::__invoke<>() at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:95
@ 0x5384d9 _ZNSt6thread8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0EEE9_M_invokeIJLm0EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:244
@ 0x538499 std::thread::_Invoker<>::operator()() at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:253
@ 0x53839b std::thread::_State_impl<>::_M_run() at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:196
@ 0x7fe3d58d6a4f execute_native_thread_routine at ??:0
@ 0x7fe3d0f8f6db start_thread at ??:0
@ 0x7fe3ca26961f clone at ??:0